Data Orchestration: Modernizing Legacy Data Orchestration with Cloud Composer

Gautami Nadkarni

Abstract: This paper discusses the migration of data orchestration workflows from a legacy tool like Autosys to a modern, cloud - based solution, Google Cloud Composer. It explores the transition from traditional job scheduling to Directed Acyclic Graph (DAG) - based workflows using Apache Airflow, culminating in the deployment and management of these workflows in Cloud Composer. The benefits and challenges of this migration are examined, highlighting the advantages of scalability, flexibility, and cloud integration offered by Cloud Composer.

Keywords: Data Orchestration, Autosys, Apache Airflow, Cloud Composer, DAG, Migration, Cloud Computing, open source