Audio-Textual Emotion Recognition using Pre-trained models: Investigating Various Representations and Fusion Techniques

Abstract Emotion recognition is vital in Human-Computer Interaction, enhancing artificial intelligence with emotional intelligence. Given the multimodal nature of human conversations, emotions can be detected through various modalities, leading to more accurate recognition. This makes multimodal emotion recognition a popular yet challenging research area. In this study, we focused on recognizing emotions from both audio and text modalities on the IEMOCAP dataset. We utilized transfer learning and fine-tuned transformer models for each modality, aiming to minimize the number of trainable parameters in the final system. One of the main challenges in multimodal emotion recognition lies in effectively fusing different modalities. To address this, we employed early fusion, cross-modal fusion, and late fusion techniques to integrate information from the audio and text models, using representations extracted from different layers of each model. Our results indicate that using the average of mean pooling across all layers for each modality, combined with an early fusion approach and a support vector machine (SVM) classifier, achieved the best performance. This approach resulted in an unweighted average recall (UAR) of 78.42%, a weighted average recall (WAR) of 77.75%, and a cross-entropy loss of 0.67, outperforming previous studies2.