Low Latency Data Lookups in Real - Time Data Engineering Pipelines

Girish Ganachari

Abstract: Healthcare, finance, and the Internet of Things need real - time data engineering pipelines with low - latency data lookups for processing and decision - making. Incremental ETL pipeline scheduling, FPGA - based hardware acceleration, and on - GPU thread - data remapping minimise latency in this research. It covers low - latency data pipeline applications, benefits, drawbacks, and prospects. It improves flexibility, productivity, and decision - making. To improve real - time data pipelines, the research proposes resource management, hardware - software interaction, and data consistency.

Keywords: Low - latency, Real - time data, Data engineering pipelines, FPGA acceleration, Incremental ETL, On - GPU processing, Data consistency

How to Cite?: Girish Ganachari, "Low Latency Data Lookups in Real - Time Data Engineering Pipelines", Volume 9 Issue 10, October 2020, International Journal of Science and Research (IJSR), Pages: 1815-1819, https://www.ijsr.net/getabstract.php?paperid=SR24801075034, DOI: https://dx.doi.org/10.21275/SR24801075034