Strategic Data Pipeline Design: Enhancing Operational Efficiency from Oracle to Single Store using Airflow S3 Data Pipelines

Pankaj Dureja

Abstract: This paper investigates an innovative ETL pipeline managed by Apache Airflow, integrating Oracle databases with SingleStore through Amazon S3. The architecture enhances efficiency, scalability, and reliability of data integration processes. By implementing a sequence of orchestrated tasks, the study demonstrates improvements in data throughput and process automation compared to traditional ETL techniques. This article is significant as it addresses the need for scalable and efficient ETL processes in enterprise data management, demonstrating the potential improvements over traditional methods.

Keywords: Data Integration, ETL, Apache Airflow, Oracle Database, Oracle Functions, Oracle External Directories, SingleStore Pipelines, SingleStore Procedure, Amazon S3 Storage, Data Transformation