Real-Time Stream Processing of Big Data

Denis Patrick Bell, Eliasu Tambominyi, Yang Chunting

Abstract: The evolution of technology has enabled the continuous generation of massive data from connected devices and sensors. As, more data becomes available, organizations are using cutting-edge tools and techniques to extract useful insight from the data immediately they are generated. Therefore, the enterprise systems are evolving and shifting towards real-time systems. So, there is a growing need for well-developed fault tolerant distributed scalable systems to handle this change with low latency. Recent years have seen the emergence of several distributed systems whose popularity is as a result of the growing demand to efficiently analyze and interpret voluminous data. This article will discuss distributed real-time stream processing of big data, the two main real-time big data processing architectures: Lambda and Kappa and the popular frameworks used for processing real time big data.

Keywords: Real-time data processing, Big data, Batch processing, Lambda & Kappa architecture