Multimodal emotion feature extraction and information fusion methods for video content

Multimodal sentiment analysis remains challenging due to the difficulty of fusing heterogeneous data like facial expressions, speech, and pose.Unlike unimodal analysis, it is often hindered by problems of information redundancy, heterogeneity, and complex temporal dynamics.This paper proposes MERCAT, a Transformer-based model featuring a cross-modal self-attention mechanism to capture deep correlations between different modalities.This design enables highly efficient and context-aware inter-modal fusion.Extensive experiments on multiple benchmarks show that MERCAT achieves excellent performance.It notably excels in emotion classification, significantly improving accuracy and F1-score over strong baselines, and in emotion intensity prediction, where it substantially reduces error and improves correlation.The study conclusively verifies the efficacy of the cross-modal attention mechanism for information fusion, providing a robust and effective solution for advancing multimodal sentiment analysis.