Cross-Attention in Audio–Text Multimodal Systems for Emotion Recognition Tasks Using the MSP-Podcast Corpus

Automatic recognition of emotions in speech (known as speech emotion recognition or SER) is fundamental in human–machine interaction. It has various applications in areas such as customer service and intelligent systems. Although there have been recent advances, it is still challenging to properly recognize emotions in real-life situations. This work presents an innovative approach that uses cross-attention mechanisms with the aim of improving performance in the SER task and studying in depth how information from textual/audio modalities interacts in recognizing emotions. All the experiments presented in this paper follow the guidelines of the emotion recognition challenge in the InterSpeech 2025 conference, one of the latest emotion recognition challenges. The MSP-Podcast corpus, which was used in the present evaluation, is one of the most recent and comprehensive emotion recognition corpora. The obtained results are comparable to those of the system that achieved the best results in the challenge using the same audio and text encoders, suggesting that cross-attentional mechanisms may capture emotional information more effectively and offer a promising direction for developing robust and generalizable SER systems.