International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064

Downloads: 7 | Views: 32 | Weekly Hits: ⮙4 | Monthly Hits: ⮙5

Informative Article | Data & Knowledge Engineering | India | Volume 12 Issue 5, May 2023 | Rating: 5.5 / 10

Empowering AI with Efficient Data Pipelines: A Python Library for Seamless Elasticsearch to BigQuery Integration

Preyaa Atri [7]

Abstract: This paper introduces a Python library designed to accelerate AI and data engineering workflows by facilitating seamless data transfer between Elasticsearch, a powerful search engine for unstructured data, and BigQuery, a scalable data warehouse platform from Google Cloud. By automating the migration of large datasets from Elasticsearch to BigQuery, the library empowers AI researchers, data scientists, and engineers to efficiently leverage cloud-based resources for model training, preprocessing, analysis, and reporting. This research delves into the library's features, dependencies, usage patterns, and its potential to enhance data management efficiency in AI-driven projects and data engineering pipelines. Additionally, the paper discusses the library's limitations and proposes future enhancements to further streamline AI development and data engineering workflows.

Keywords: Data Migration, Elasticsearch, BigQuery, AI, Data Engineering, Python Library

Edition: Volume 12 Issue 5, May 2023,

Pages: 2664 - 2666

How to Download this Article?

Type Your Valid Email Address below to Receive the Article PDF Link

Verification Code will appear in 2 Seconds ... Wait