Strategic Data Pipeline Design: Enhancing Operational Efficiency from Oracle to Single Store using Airflow S3 Data Pipelines
International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064


Downloads: 10 | Views: 248 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Informative Article | Information Technology | United States of America | Volume 13 Issue 8, August 2024 | Popularity: 5.5 / 10


     

Strategic Data Pipeline Design: Enhancing Operational Efficiency from Oracle to Single Store using Airflow S3 Data Pipelines

Pankaj Dureja


Abstract: This paper investigates an innovative ETL pipeline managed by Apache Airflow, integrating Oracle databases with SingleStore through Amazon S3. The architecture enhances efficiency, scalability, and reliability of data integration processes. By implementing a sequence of orchestrated tasks, the study demonstrates improvements in data throughput and process automation compared to traditional ETL techniques. This article is significant as it addresses the need for scalable and efficient ETL processes in enterprise data management, demonstrating the potential improvements over traditional methods.


Keywords: Data Integration, ETL, Apache Airflow, Oracle Database, Oracle Functions, Oracle External Directories, SingleStore Pipelines, SingleStore Procedure, Amazon S3 Storage, Data Transformation


Edition: Volume 13 Issue 8, August 2024


Pages: 285 - 290


DOI: https://www.doi.org/10.21275/SR24731115303



Make Sure to Disable the Pop-Up Blocker of Web Browser


Text copied to Clipboard!
Pankaj Dureja, "Strategic Data Pipeline Design: Enhancing Operational Efficiency from Oracle to Single Store using Airflow S3 Data Pipelines", International Journal of Science and Research (IJSR), Volume 13 Issue 8, August 2024, pp. 285-290, https://www.ijsr.net/getabstract.php?paperid=SR24731115303, DOI: https://www.doi.org/10.21275/SR24731115303

Similar Articles

Downloads: 1 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Research Paper, Information Technology, India, Volume 11 Issue 3, March 2022

Pages: 1642 - 1649

Data Integration Strategies in Hybrid Cloud Environments

Sai Kumar Reddy Thumburu

Share this Article

Downloads: 1 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Research Paper, Information Technology, United States of America, Volume 13 Issue 12, December 2024

Pages: 1348 - 1357

Autonomous Claims Processing: Building Self-Driving Workflows with Gen AI and ML in Guidewire

Sateesh Reddy Adavelli

Share this Article

Downloads: 2 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Analysis Study Research Paper, Information Technology, United States of America, Volume 11 Issue 1, January 2022

Pages: 1647 - 1652

Data Engineering Challenges in AI for Healthcare

Nithin Reddy Desani, Srujan Reddy Jabbi Reddy

Share this Article

Downloads: 2 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Research Paper, Information Technology, India, Volume 9 Issue 12, December 2020

Pages: 1866 - 1874

Data Vault 2.0: Evolution and Adoption in Large Enterprises

Guruprasad Nookala

Share this Article

Downloads: 3 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Informative Article, Information Technology, India, Volume 12 Issue 12, December 2023

Pages: 2100 - 2104

Bridging Data Silos: Enhancing Business Operations through Advanced Data Integration and System Interoperability

Sree Sandhya Kona

Share this Article
Top