Ulcr: Unimodal Label Generation Strategy Using Contrastive Recalibrated Representations for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) utilizes multimodal representations to understand people's emotional states and predict defined emotional categories. Effective modality representations should encompass two characteristics - consistency and complementarity. Existing methods typically treat representations from different modalities with equal weights, ignoring the disparate contributions they make. In addition, existing MSA benchmark datasets lack unimodal annotations to guide the extraction of contradictory signals from each modality, which has primarily focused on constructing multimodal representations. We propose a Cross-modal Language perception Network (CLN), which leverages language representation as the core to extract and fuse acoustic and visual signals, aiming to achieve consistency in multimodal representation. To handle the challenge from absence of unimodal annotations, we introduce Unimodal Label generation strategy using Contrastive Recalibrated modality representations (ULCR) to acquire more accurate label-shifted data by recalibrating three unimodal representations through contrastive learning. We perform joint training on multimodal and unimodal tasks to understand the consistency and distinctiveness of modal representations separately. Experiments on three public datasets show that the performance of our proposed model surpasses that of the state-of-the-art baselines. The full codes are available at https://github.com/115239/ULCR.