Autonomous Scheduling for Recurring Tasks to Manage Ingestion and Stream Processing

Mahidhar Mullapudi, Satish Kathiriya, Siva Karthik Devineni

Abstract: The need for effective large-scale heterogeneous distributed data ingestion pipelines, crucial for transforming and processing data essential to advanced analytics and machine learning models, has seen a significant surge in importance. Modern services increasingly depend on near-real-time signals to precisely identify or predict customer behavior, sentiments, and anomalies, thereby facilitating informed, data-driven decision-making.In the rapidly evolving landscape of large-scale enterprise data applications, the demand for efficient data ingestion and stream processing solutions has never been more critical. This technical paper introduces a groundbreaking autonomous self-schedulable library designed for recurring jobs, addressing the challenges faced by enterprises in orchestrating complex data workflows seamlessly. Leveraging authoritative expertise in building robust enterprise applications, this library provides a paradigm shift in how organizations manage and execute recurring tasks within data pipelines.

Keywords: Modern Ingestion Platform, Autonomous Scheduling Library, Parallel Stream Processing

How to Cite?: Mahidhar Mullapudi, Satish Kathiriya, Siva Karthik Devineni, "Autonomous Scheduling for Recurring Tasks to Manage Ingestion and Stream Processing", Volume 10 Issue 9, September 2021, International Journal of Science and Research (IJSR), Pages: 1731-1736, https://www.ijsr.net/getabstract.php?paperid=SR24203215713, DOI: https://dx.doi.org/10.21275/SR24203215713