Emotion recognition in financial applications using multimodal natural language processing and dual-encoder cross-modal attention

Financial markets are significantly influenced by investor psychology, yet most existing sentiment analysis systems classify opinions into binary categories (bullish/bearish), failing to capture the nuanced emotional states—such as anxiety, panic, optimism, and excitement—that may be associated with market behavior. This study evaluates a multimodal financial emotion-recognition framework, termed the Unified Multimodal Transformer (UMT), implemented as a dual-encoder cross-modal attention model. The system integrates investor text, emoji and cashtag cues, and historical price sequences for binary sentiment classification, 12-class emotion classification, and short-horizon return forecasting. UMT is used as the study-specific name of the integrated system and does not represent a claim of a fundamentally new Transformer backbone. Experiments used the publicly available StockEmotions dataset containing 10,000 English StockTwits comments collected from January to December 2020. The repository-provided random split of 8,000 training, 1,000 validation, and 1,000 test samples was used for the classification tasks. For forecasting, each post was paired only with price observations available before its prediction timestamp, and forecasting evaluation was conducted using chronologically separated samples. The framework combined DistilBERT, a temporal convolutional price encoder, bidirectional cross-modal attention, gated fusion, and multi-task learning. Experiments were repeated across five random seeds. UMT achieved a weighted sentiment F1-score of 0.841 (95% CI [0.831, 0.851]) and a weighted emotion F1-score of 0.461. In the forecasting evaluation, UMT obtained an MSE of 5.61 × 10⁻⁴, representing a 33.4% reduction relative to the price-only LSTM baseline. A descriptive attention diagnostic assigned 32.3% of normalized attention to anxiety and panic and 27.9% to excitement and optimism. This difference describes model feature allocation within the 2020 sample and should not be interpreted as a causal or psychological effect estimate. Within the 2020 COVID-era sample, integrating investor text, fine-grained emotion supervision, and lagged price information improved the evaluated prediction tasks relative to the controlled baselines used in this study. The fear–joy attention difference is a model-specific predictive association and should not be interpreted as evidence that fear causes market movements or as a direct test of Prospect Theory. Validation using independent post-2020 market regimes remains necessary.