Answers are grounded in this paper and cite their sources.
Audio-Visual Emotion Recognition (AVER) from speech and facial expression is a fundamental task in affective computing, yet effectively fusing multimodal features while accounting for both modality consistency and discrepancy remains a challenging problem. Existing approaches either neglect modality discrepancy or model consistency and discrepancy in separate feature spaces. This paper proposes a Consistency-Constrained Fusion Network (CCFNet) that jointly captures unimodal consistency and discrepancy within a unified architecture. To extract modality consistency, two complementary feature fusion models are designed: an attention-based model that performs unimodal self-fusion followed by multimodal cross-fusion, and a Feature-wise Linear Modulation (FiLM) based model that uses facial expression features to modulate speech channel features. To retain modality discrepancy without performance degradation, a consistency encoder constrained by a Central Moment Discrepancy (CMD) loss maps unimodal features into a shared vector space, allowing discrepant features to be preserved rather than discarded as noise. A gate-structure based model aggregates segment-level fused features into video-level representations, and a two-stage training strategy addresses the different learning dynamics between modalities. Experiments on the RAVDESS and CMU-MOSEI datasets demonstrate the effectiveness of the proposed method, achieving recognition accuracies of 79.03% and 68.27% (weighted accuracy), respectively. Ablation studies confirm that each component of the framework contributes to the overall performance.
No takes yet. Share an insight, caveat, or question.
Liang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: