Continuously self-adaptive multimodal sentiment analysis

Purpose This paper aims to address modality noise and continuous changing of reliabilities of emotional cues in multimodal sentiment analysis. Such dynamic perceptual characteristics are challenging for continual sentiment perception, especially when there are modality conflicts. Design/methodology/approach The authors propose a Continuously self-Adaptive Multimodal Sentiment Analysis method (CAMSA). CAMSA decouples modality features into shared and private features and introduces a Variational Information Bottleneck method to perform probabilistic modeling and information compression on shared features, and then continually suppressing redundant and noisy information during the progressively emerging of emotional cues. In addition, a multidimensional adaptive feature fusion mechanism is designed to evaluate modality reliability from the perspective of semantic certainty, modal consistency and sentiment extremity. Findings Experimental results on CMU-MOSI, CMU-MOSEI and CH-SIMS demonstrate that CAMSA outperforms existing models on most evaluation metrics, validating its effectiveness in handling modality noise and modal conflicts. Originality/value From the requirements of continual sentiment perception, this paper proposes a continuously self-adaptive CAMSA framework, which combines shared feature refinement with a multidimensional adaptive fusion mechanism, enabling robust cross-modal sentiment modeling in dynamic and evolving perceptual scenarios. The CAMSA approach can effectively address the problem of fixed fusion strategies in complex dynamic sentiment perception tasks.