International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064




Downloads: 0 | Views: 37

Informative Article | Data & Knowledge Engineering | India | Volume 12 Issue 5, May 2023 | Rating: 4.8 / 10


Leveraging Reinforcement Learning for Autonomous Data Pipeline Optimization and Management

Chandrakanth Lekkala [5]


Abstract: The growing advancement of data pipelines in large - scale data processing systems presents many problems for efficient resource management and optimization. Reinforcement Learning (RL) was developed as an efficient method to handle complex systems smartly with application in data pipeline autonomy and optimization. The essay examines implementing RL techniques to optimize data pipelines and resource distribution in big data processing systems. We discuss the problems of applying RL in data pipeline situations, like defining the reward functions and the delayed feedback. We also examine the RL algorithms like Q learning and Policy Gradients. The two sections summarise findings of recent research and case studies, in which they point out how RL can help improve data pipeline competence regarding resource usage and a quick response to changes within workloads. Lastly, we explore emerging research trends and the unsolved problems in the RL - driven data pipeline optimization area.


Keywords: Reinforcement Learning, Data Pipelines, Autonomous Optimization, Resource Management, Q - learning, Policy Gradients


Edition: Volume 12 Issue 5, May 2023,


Pages: 2667 - 2674



How to Download this Article?

Type Your Valid Email Address below to Receive the Article PDF Link


Verification Code will appear in 2 Seconds ... Wait

Top