AI, ML Governance and Feature Engineering on Lakehouses: Unifying Data Engineering and ML Engineering with Azure Databricks and Unity Catalog

Amol Bhatnagar

Abstract: The convergence of data engineering and machine learning engineering has become imperative as organizations scale their AI initiatives. Traditional architectures that separate analytical data platforms from ML infrastructure create operational silos, governance gaps, and inefficient workflows. The lakehouse paradigm, exemplified by Azure Databricks with Unity Catalog, addresses these challenges by providing a unified platform for both analytics and machine learning. This paper examines how modern lakehouses enable seamless integration of data engineering and ML engineering through shared storage formats, centralized governance, and comprehensive lineage tracking. We explore the architecture of feature stores built on open table formats like Delta Lake, the governance capabilities of Unity Catalog for ML datasets and models, the role of MLflow in managing the ML lifecycle, end-to-end lineage tracking from raw data to deployed models, and role-based access control for secure model sharing. Through detailed analysis of Azure Databricks capabilities, we demonstrate how organizations can establish robust ML governance frameworks while maintaining the agility required for rapid experimentation and deployment. Our findings indicate that unified lakehouse platforms reduce time-to-production for ML models by 40-60% while simultaneously improving governance compliance and model quality.

Keywords: lakehouse architecture, machine learning governance, data engineering integration, feature stores, ML lifecycle management

How to Cite?: Amol Bhatnagar, "AI, ML Governance and Feature Engineering on Lakehouses: Unifying Data Engineering and ML Engineering with Azure Databricks and Unity Catalog", Volume 15 Issue 2, February 2026, International Journal of Science and Research (IJSR), Pages: 48-57, https://www.ijsr.net/getabstract.php?paperid=SR26130120835, DOI: https://dx.doi.org/10.21275/SR26130120835