Implementing HACE Theorem for Big Data Processing

Prema Gadling, Mahip Bartere

Abstract: The aim is propose to elaborate a HACE theorem that states the characteristics of the Big Data revolution, and proposes a Big Data processing model from the data mining view. Here, Data comes from everywhere like sensors, media sites and social media etc. In this useful data can be extracted from this big data using data mining technique for discovering interesting patterns. As enhancement we propose Detection of emerging topics from social networks of big data. Specifically, we focus on mentions of user links between users that are generated dynamically (intentionally or unintentionally) through replies, mentions, and retweets. In this paper, we are going to talk how effectively analysis is done on the data which is collected from the Twitter using Flume. Twitter is an online web application which contains rich amount of data that can be a structured, semi-structured and un-structured data. We can collect the data from the twitter by using big data eco-system using online streaming tool Flume. And doing analysis on Twitter is also difficult due to language that is used for comments. And, coming to analysis there are different types of analysis that can be done on the collected data. And this paper provides a way of analysis data using hadoop which will process the huge amount of data on a hadoop cluster faster in real time.

Keywords: Big Data, data mining, REST API, HACE THEOREM, HDFS, LDA, Sentiment analysis