This is the authors' abstract. We don't add key points for this paper.
In recent years, multimodal emotion recognition (MER) has gained significant attention due to its potential to integrate information from diverse signals. However, existing methods often struggle to effectively capture complex interactions and contextual information both inter- and intra-modalities, and even to extract the salient representations from pre-trained models. To address these issues, we propose a novel model, gated Mixture of Multimodal Experts (MoME) and Mixtral of Experts (MixMoE) models, namely GateM2Former. The gate mechanism is used to select the most relevant representations from pre-trained models. The MoME and MixMoE expert modules respectively learn the individual characteristics of each modality and the intrinsic alignment and potential interactions between modalities. Besides, we design a hierarchical merge structure to better suit the long sequence scenario (i. e., speech in our case). To verify the effectiveness of the introduced model, we conducted extensive experiments on the IEMOCAP and MELD datasets. The results show that GateM2Former, with a universal multimodal structure, is able to achieve the best results on IEMOCAP and MELD compared with other latest approaches.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Xu et al. (2025) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: