Multimodal Insights into Granger Causality Connectivity: Integrating Physiological Signals and Gated Eye-Tracking Data for Emotion Recognition Using Convolutional Neural Network

Highlights•Introduces ECG-EEG integration with eye-tracking for emotion recognition.•Employs pupil diameter metrics to refine emotional data accuracy.•Innovative Granger causality approach to study brain-heart interaction.•Utilizes ResNet-18 for high accuracy: 91 % and AUC of 0.97.•Outperforms state-of-the-art with eye-tracking data enhancement.AbstractThis study introduces a groundbreaking method to enhance the accuracy and reliability of emotion recognition systems by combining electrocardiogram (ECG) with electroencephalogram (EEG) data, using an eye-tracking gated strategy. Initially, we propose a technique to filter out irrelevant portions of emotional data by employing pupil diameter metrics from eye-tracking data. Subsequently, we introduce an innovative approach for estimating effective connectivity to capture the dynamic interaction between the brain and the heart during emotional states of happiness and sadness. Granger causality (GC) is estimated and utilized to optimize input for a highly effective pre-trained convolutional neural network (CNN), specifically ResNet-18. To assess this methodology, we employed EEG and ECG data from the publicly available MAHNOB-HCI database, using a 5-fold cross-validation approach. Our method achieved an impressive average accuracy and area under the curve (AUC) of 91.00 % and 0.97, respectively, for GC-EEG-ECG images processed with ResNet-18. Comparative analysis with state-of-the-art studies clearly shows that augmenting ECG with EEG and refining data with an eye-tracking strategy significantly enhances emotion recognition performance across various emotions.