Harnessing Big Data and NLP for Real-Time Market Sentiment Analysis Across Global News and Social Media

Rashi Nimesh Kumar Dhenia

Abstract: This research explores a scalable pipeline that utilizes Natural Language Processing (NLP) and Big Data technologies for real-time market sentiment analysis across global news and social media platforms. The exponential growth of unstructured content, ranging from financial headlines to social media opinions, presents an opportunity for businesses to extract actionable intelligence. We propose a hybrid framework that integrates web scraping, text preprocessing, sentiment scoring (using VADER, TextBlob, and BERT), topic modeling (LDA, BERTopic), and real time dashboards via Apache Kafka, Spark, and Power BI. This system supports businesses and financial analysts with continuous, contextual, and high-resolution sentiment streams. The solution?s scalability, multilingual support, and high accuracy demonstrate its readiness for real world deployment.

Keywords: Big Data, Data Analysis, Natural Language Processing, Sentiment Analysis, Real-Time Analytics, Market Intelligence, Topic Modeling, Stream Processing, Power BI