Data Mining and Uncertainty-Aware with Missing Modalities for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) integrates diverse modalities to identify emotional states, yet performance often suffers in scenarios with missing data. In this situation, despite the promising results of recent methods, the failure of part methods to fully exploit the latent valid information contained in incomplete modalities may degrade predictive performance. Besides, to address the oversight of varying contributions across modalities to sentiment understanding, the score-based weighting schemes in the exhibited methods remain overly sensitive to data fluctuations, leading to unstable and unreliable predictions. To this end, we propose a novel method, Data Mining and Uncertainty-Aware with Missing Modalities for Multimodal Sentiment Analysis (DUDF-MSA). In this method, the Common Feature Extraction (CFE) mechanism is introduced to learn common features across modalities, thereby guiding attention toward critical cues in the missing modalities. In parallel, the specific feature uncertainty-aware dynamic adjustment (SFUA) scheme is designed to, in addition to extracting modality-specific features, adaptively assess each modality’s contribution by quantifying its class uncertainty based on the probability distribution of its features. According to these weights, our method can effectively mitigate the negative impact of ambiguous sentiment cues from unreliable modalities during the following feature aggregation stage. Finally, the common features and the adjusted modality-specific features are jointly learned to predict sentiment intensity. The experiments conducted on benchmark datasets indicate the superior performance of DUDF-MSA, which yielding 34.29% Acc-7 (1.062 MAE) on MOSI and 35.66% Acc-5 (0.505 MAE) on SIMS.