A Multitask Framework for Emotion Recognition Using EEG and Eye Movement Signals with Adversarial Training and Attention Mechanism

Affective brain-computer interface is one of the research frontiers which could promote the development of artificial intelligence and which might help the diagnosis and treatment of mental health diseases. In the field of emotion recognition with EEG and eye movement signals, however, it is challenging to build a model that can extract emotion-related information, fuse multiple modalities, and preserve the complementary characteristics at the same time. In this paper, we proposed a multi-task framework with adversarial training and attention mechanism (ATAM) for emotion recognition using EEG and eye movement signals. An adversarial training scheme is designed by maximizing mutual information loss within the same modalities and minimizing the cosine similarity loss between different modalities. With this design, the proposed ATAM not only preserved emotional information in EEG and eye movement signals, but also kept the complementary property between these two modalities. Attention mechanism was used to fuse multimodal features adaptively. Our evaluation of the model using the SEED and SEED-IV datasets revealed that the proposed method surpassed existing methodologies in recognition accuracy. Further analysis of the loss curves and attentional weight distributions indicated the effectiveness of ATAM in transforming and incorporating multi-modal properties. The adversarial training set-up, epitomized in our ablation study, was necessary for good performance.