International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064

Downloads: 0 | Views: 37

Informative Article | Data & Knowledge Engineering | India | Volume 12 Issue 5, May 2023 | Rating: 4.8 / 10

Leveraging Reinforcement Learning for Autonomous Data Pipeline Optimization and Management

Chandrakanth Lekkala [5]

Abstract: The growing advancement of data pipelines in large - scale data processing systems presents many problems for efficient resource management and optimization. Reinforcement Learning (RL) was developed as an efficient method to handle complex systems smartly with application in data pipeline autonomy and optimization. The essay examines implementing RL techniques to optimize data pipelines and resource distribution in big data processing systems. We discuss the problems of applying RL in data pipeline situations, like defining the reward functions and the delayed feedback. We also examine the RL algorithms like Q learning and Policy Gradients. The two sections summarise findings of recent research and case studies, in which they point out how RL can help improve data pipeline competence regarding resource usage and a quick response to changes within workloads. Lastly, we explore emerging research trends and the unsolved problems in the RL - driven data pipeline optimization area.

Keywords: Reinforcement Learning, Data Pipelines, Autonomous Optimization, Resource Management, Q - learning, Policy Gradients

Edition: Volume 12 Issue 5, May 2023,

Pages: 2667 - 2674

How to Download this Article?

Type Your Valid Email Address below to Receive the Article PDF Link

Verification Code will appear in 2 Seconds ... Wait