AdaMoW: Multimodal Sentiment Analysis Based on Adaptive Modality-Specific Weight Fusion Network

Multimodal sentiment analysis (MSA) is a crucial task in the field of natural language processing (NLP), with a wide range of applications. This paper proposes an adaptive modality-specific weight fusion network (AdaMoW) to address issues in the process of multimodal data fusion. Specifically, we use different weight calculation methods at various stages of the model. In the model training stage, diverse weights are assigned to different modalities by calculating the correlation between the single-modal sentiment prediction value and the real multimodal sentiment labels, and a weight-mapping network is designed to learn this “data-weight” mapping relationship. In the testing and verification phase of the model, the trained weight-mapping network is used to obtain the weights of different modalities. In addition, in order to optimize the multimodal fusion data, we designed a generator, which reversely generates the unimodal feature vector through the multimodal fusion vector, and compares it with the original unimodal feature extraction obtained after unimodal feature extraction. The modal feature vectors are compared and optimized, so that the fusion results can maintain the uniqueness of the modality while obtaining the multimodal data interaction information. AdaMoW is verified on two benchmark MSA datasets CMU-MOSI and CMU-MOSEI. The experimental results show that the effectiveness of AdaMoW surpasses the previous baseline model and achieves state-of-the-art results.