Hybrid-Module Transformer: enhancing speech emotion recognition with HuBERT, LSTM, and ResNet-50

Speech emotion recognition (SER) is a challenging task that involves identifying human emotions from speech. Traditional sequence models like recurrent neural network (RNN) and long short-term memory (LSTM) are limited by vanishing gradients and difficulty in capturing long-range dependencies. This article presents a novel model based on the Hybrid-Module-Transformer, which leverages the capabilities of Transformer modules to extract feature representations effectively, even with limited data. The model combines the strengths of Hidden-Unit BERT (HuBERT), LSTM, and Residual Network (ResNet-50) to achieve superior performance in speech emotion classification tasks. In the model, we utilized Mel-frequency cepstral coefficients (MFCC) and Spectrogram for feature extraction. Then, a HuBERT-LSTM framework is used to perform both speech-to-text recognition and emotion classification. We evaluate the model on two benchmark datasets: Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and Multimodal EmotionLines Dataset (MELD). On the RAVDESS dataset, the model achieves a maximum accuracy of 76% and precision of 78%, while on the more challenging MELD dataset, it attains an accuracy of 72.9% and precision of 72.3%. These results demonstrate the effectiveness and generalizability of our model in both controlled and real-world conversational scenarios, making it a competitive solution for robust speech emotion recognition.