Robust Speech Emotion Recognition Through Self-Supervised Acoustic-Semantic Fusion and Cross-Attention Under Background Noise

Human speech carries information about both the words being spoken and the emotions behind them. Speech Emotion Recognition (SER) uses artificial intelligence to identify emotions such as happiness, sadness, anger, and neutrality from a person's voice. However, recognizing emotions accurately is difficult when speech contains background noise or when different people express the same emotion in different ways. This paper proposes a speech emotion recognition framework that combines self-supervised speech representations with acoustic features through cross-attention. A pretrained speech encoder is used to extract useful representations from audio, while acoustic features such as pitch, energy, MFCCs, and spectral information provide additional details about vocal expression. The two feature streams are combined through a cross-attention mechanism before the final emotion classification stage. The main aim is to investigate whether this combination can improve emotion recognition under different background-noise conditions. The proposed system will be evaluated on a publicly available emotional speech dataset and compared with acoustic-only, pretrained-representation-only, and conventional feature-fusion methods. Accuracy, unweighted accuracy, macro-F1 score, and class-wise recall will be used for evaluation. The experimental results will determine whether the proposed method offers a meaningful improvement over existing approaches.