Downloads: 17 | Views: 263 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1
Comparative Studies | Science and Technology | United States of America | Volume 13 Issue 9, September 2024 | Rating: 6 / 10
Integrating Databricks with CI/CD Pipelines for Continuous Data Engineering
Satyadeepak Bollineni
Abstract: Big data processing and analytics tool Databricks being brought into the Continuous Integration and Continuous Deployment (CI/CD) pipeline are quickly becoming the industry standard in data engineering. This paper discusses the possibility and advantages of such integration while focusing on how Data bricks can improve and streamline data engineering steps with CI/CD. Bridging these areas with this integration, their associated benefits, implementation strategies, risks and rewards, and implications for data management and efficiency are explored to provide insights into practical data engineering.
Keywords: Databricks CI/CD Continuous Integration, Continuous Deployment, Data Engineering, Automation, Big Data, DevOps, Workflow Automation, Pipeline Integration
Edition: Volume 13 Issue 9, September 2024,
Pages: 733 - 738