Cross-modal Synergy for Enhancing Emotion Recognition Through Integrated Audio–Video Fusion Techniques

Abstract Emotion identification is important for human–computer interaction, and its applications include medical and customer service provision. In the past, emotion recognition systems have been based on one modality, such as text, image, audio, or video, each with advantages. However, these single-mode approaches are often less accurate due to their failure to fully capture the complexity of human emotions. Several limitations characterize the existing methods, such as isolated analysis per modality, loss of contextual information, and sub-optimal performance. Usually, these approaches cannot accurately detect emotions in real-life situations where multiple channels exist through which people express themselves in terms of experiencing feelings. To overcome this situation,a multi-modal emotional recognition using a cross-modal fusion (MER-CMF) system is proposed that makes different data sources work together to understand emotional states more completely and holistically. The fusion process synergizes the strengths of each modality, thus enabling the system to capture subtle nuances in emotional expressions. Consequently, the MER-CMF model enhances emotion recognition accuracy significantly compared to existing methods. Our experimental results show that it outperforms unimodal baselines in terms of accuracy and F1 score, achieving an impressive accuracy of 98.32%. These results demonstrate that cross-modality fusion can improve the reliability and efficiency of emotion recognition systems so that they can become more intuitive and responsive for their users.