Emotion recognition based on the temporal patterns of electroencephalogram signals and electrodermal response signals using the TRANSFORMER network

Figures

• 

  FIGURE 1. Experimental principle diagram.

• 

  FIGURE 2. Overall training convergence and cross-validation performance of the proposed multimodal Transformer framework. (a) Training and validation accuracy curves across epochs for the proposed model and the Bi-LSTM baseline. (b) Fold-wise comparison of accuracy, precision, recall, and F1-score under subject-independent five-fold cross-validation.

• 

  FIGURE 3. Detailed classification evaluation of the proposed multimodal Transformer model. (a) Confusion matrix for four-class emotion recognition. (b) Receiver operating characteristic (ROC) curves and area under the curve (AUC) values for each emotional category. (c) Average precision, recall, and F1-score with standard deviation error bars across folds. (d) Distribution of class-wise accuracy represented using box plots.

• 

  FIGURE 4. Comparative performance analysis of unimodal and multimodal emotion recognition models. (a) Emotion recognition accuracy obtained using GSR-only, EEG-only, and combined EEG-GSR inputs across emotional categories. (b) Overall classification accuracy comparison among different deep learning architectures, including CNN, Bi-LSTM, and the proposed Transformer framework.

• 

  FIGURE 5. Visualization and interpretability analysis of the proposed multimodal Transformer framework. (a) t-SNE visualization of learned emotional feature distributions. (b) Self-attention weight heatmap within the Transformer encoder. (c) Radar chart comparing class-wise precision between the proposed model and the CNN baseline. (d) Relative feature contribution ratio between EEG and GSR modalities.

• 

  FIGURE 6. Statistical robustness and computational performance evaluation of the proposed framework. (a) Bland-Altman analysis of fold-wise prediction consistency. (b) Relationship between inference time and classification accuracy for different model architectures. (c) Misclassification rate analysis for major error sources observed during emotion recognition.

• 

  FIGURE 7. Representative physiological signal segments after preprocessing. (a) Preprocessed EEG signal waveform recorded from the Fp1 channel. (b) Corresponding preprocessed galvanic skin response (GSR) signal segment obtained during emotional stimulation.