Scalable Microservice-Based Data Quality Framework Using Great Expectations and BigQuery on Google Kubernetes Engine

Vidit Jain

Abstract: This paper presents a comprehensive data quality frame-work implemented as a microservice architecture on Google Kubernetes Engine (GKE). The framework leverages Great Expectations for data validation and BigQuery for efficient data processing, ensuring high data quality across diverse data pipelines. Comparative analysis with leading data quality solutions demonstrates significant improvements in scalability (40% better throughput) and cost-efficiency (35% lower processing costs). Our architecture supports both batch and near real-time validation with measured latency under 30 seconds for streaming workflows. Implementation at a large financial institution resulted in a 78% reduction in data quality incidents. The empirical evaluation confirms the framework?s effectiveness across varying workloads while maintaining security and governance standards required in enterprise environments.

Keywords: Data Quality, Microservices, Cloud Computing, Google Kubernetes Engine, BigQuery, Great Expectations