The multimodal Transformer framework achieved a mean classification accuracy of 87.42% for four-class emotion recognition, outperforming CNN and Bi-LSTM baselines by 4.91% and 6.38%, respectively.
Does a Transformer-based multimodal framework improve emotion recognition accuracy using EEG and GSR signals in neurologically healthy adults?
A Transformer-based multimodal framework jointly modeling EEG and GSR signals significantly improves emotion recognition accuracy compared to traditional baselines.
Introduction Emotion recognition using physiological signals plays an important role in affective neuroscience and human-centered artificial intelligence. Current methods still face challenges in long-range temporal dependency modeling and explicit central–autonomic coupling representation, while generalization under subject-independent protocols needs further improvement. Methods This study proposes a Transformer-based multimodal framework for four-class discrete emotion recognition (neutral, happiness, sadness, and fear) by jointly modeling EEG and GSR signals. The architecture integrates temporal self-attention and bidirectional cross-modal attention. Experiments were conducted on 42 neurologically healthy adults with a controlled audiovisual emotion elicitation paradigm, evaluated using subject-independent five-fold cross-validation. Results The model achieved a mean classification accuracy of 87.42% ± 2.13%, with precision of 87.6%, recall of 87.4%, and F1-score of 87.5%. It outperformed CNN and Bi-LSTM baselines by 4.91% and 6.38%, respectively. Multimodal fusion significantly boosted high-arousal emotion recognition, with fear accuracy increasing from 82.11% (EEG-only) to 88.63% ( p = 0.004). Discussion These findings confirm that long-range temporal modeling and explicit cross-modal interaction can substantially improve multimodal physiological emotion recognition. The proposed framework is scalable and interpretable, advances central–autonomic coupling modeling, enhances generalization via strict subject-independent validation, and supports physiological interpretability through attention visualization and modality sensitivity analysis.
Z Z (Fri,) conducted a other in Healthy adults (Emotion recognition) (n=42). Multimodal Transformer framework (EEG and GSR) vs. CNN and Bi-LSTM baselines was evaluated on Mean classification accuracy. The multimodal Transformer framework achieved a mean classification accuracy of 87.42% for four-class emotion recognition, outperforming CNN and Bi-LSTM baselines by 4.91% and 6.38%, respectively.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: