AI driven sentiment analysis in financial markets: using transformer base models and social media signals for stock market predictions

Purpose This study aims to explore the predictive role of artificial intelligence (AI)-driven sentiment analysis in financial markets by developing a hybrid long short-term memory–Random Forest framework. It investigates whether the integration of generative sentiment signals with historical market data can enhance the accuracy and robustness of stock price forecasting and financial predictions across various industry sectors. Design/methodology/approach This research uses a multisource data set from 2019 to 2024, including stock price data from Yahoo Finance, macroeconomic indicators from Federal Reserve Economic Data and textual sentiment from Reddit, Twitter, Bloomberg and Reuters. Transformer-based natural language processing models, such as FinBERT, are used to quantify sentiment, which is then used as a predictive feature in machine learning models. Granger causality analysis and accuracy metrics are applied to evaluate sectoral variations in sentiment impact. Findings Empirical analysis reveals that social media sentiment Granger causes short-term stock movements in technology and finance sectors, with the hybrid model achieving 68.5% directional accuracy and a 22% reduction in prediction error compared to ARIMA models benchmarks. In contrast, sectors like healthcare and energy show minimal sensitivity to sentiment, underscoring the need for domain-specific strategies. This study also identifies ethical concerns related to sentiment manipulation, transparency and AI governance in financial contexts. Originality/value This research introduces a reproducible, cross-sectoral forecasting framework that bridges AI, sentiment analysis and finance. The proposed architecture offers practical forecasting enhancements and contributes to ethical discourse on AI use in high-stakes financial environments, with implications for regulators, analysts and portfolio managers.