Downloads: 4 | Views: 50 | Weekly Hits: ⮙1 | Monthly Hits: ⮙1

Informative Article | Science and Technology | India | Volume 10 Issue 9, September 2021

Autonomous Scheduling for Recurring Tasks to Manage Ingestion and Stream Processing

Mahidhar Mullapudi ^[11] | Satish Kathiriya ^[9] | Siva Karthik Devineni ^[8]

Abstract: The need for effective large-scale heterogeneous distributed data ingestion pipelines, crucial for transforming and processing data essential to advanced analytics and machine learning models, has seen a significant surge in importance. Modern services increasingly depend on near-real-time signals to precisely identify or predict customer behavior, sentiments, and anomalies, thereby facilitating informed, data-driven decision-making.In the rapidly evolving landscape of large-scale enterprise data applications, the demand for efficient data ingestion and stream processing solutions has never been more critical. This technical paper introduces a groundbreaking autonomous self-schedulable library designed for recurring jobs, addressing the challenges faced by enterprises in orchestrating complex data workflows seamlessly. Leveraging authoritative expertise in building robust enterprise applications, this library provides a paradigm shift in how organizations manage and execute recurring tasks within data pipelines.

Keywords: Modern Ingestion Platform, Autonomous Scheduling Library, Parallel Stream Processing

Edition: Volume 10 Issue 9, September 2021,

Pages: 1731 - 1736