Attention-based fusion for multimodal sentiment analysis using emotion mapping from behavioral, physiological, and textual signals

Multimodal Sentiment Analysis (MSA) involves integrating diverse data modalities—such as physiological signals, speech, and textual input—to predict human emotions with higher reliability than unimodal approaches. However, key challenges persist in modeling cross-modal interactions, handling modality-specific noise, and maintaining predictive stability when individual modalities are weak, inconsistent, or missing. Existing approaches often overlook the contextual dependencies within each modality and fail to adaptively balance their contributions during fusion, leading to poor generalization in real-world scenarios. This work proposes a deep learning framework built on hierarchical attention-based fusion, which models both intra-modal relationships and inter-modal dependencies through self-attention, cross-attention, and multi-head attention mechanisms. Feature extraction pipelines are tailored to capture spatial and temporal patterns within each modality using convolutional and recurrent layers. These features are then dynamically aligned and fused using attention-driven modules, enabling the model to selectively focus on salient signals and suppress irrelevant or noisy information. To improve robustness and generalization, the architecture incorporates an ensemble learning strategy that combines multiple fusion models—including early fusion, late fusion, gated fusion, and graph-based fusion—via validation-weighted averaging. Training is stabilized using regularization techniques such as dropout and L2 penalty, adaptive learning rate scheduling, and class imbalance handling through synthetic data augmentation. Experimental analysis demonstrates that this approach significantly enhances sentiment prediction performance, offering a scalable and resilient solution for emotion-aware systems in complex, multimodal environments.