Scalable Data Platform Architecture for Highly Variable e-Commerce Workloads

Gautham Ram Rajendiran

Abstract: The exponential growth in the generation of data related to eCommerce has motivated organizations to seek big data architectures that scale both batch and real-time data processing [1]. This paper discusses an architecture that has fault tolerance, scalability, and low latency for data processing and was implemented in a large-scale eCommerce organization. This architecture ensures integration from varied sources, features data quality, lineage, and provides robust data storage and analytics. It entails design considerations for efficient utilization of cloud-native services, event-driven processing and task orchestration. We explain in detail the various components and brief on technical implementation details so that this can be a standard architecture that can be adopted to build eCommerce data platforms, thereby saving the time and effort in researching and designing such a platform from scratch.

Keywords: Big Data, Scalability, eCommerce, Data Architecture, Data Processing, Distributed Systems, Cloud Computing, Data Pipelines