Neural correlates of affective context in facial expression analysis: A simultaneous EEG-fNIRS study

We present a framework to measure the correlation between spontaneous human facial affective expressions and relevant brain activity. The affective states were registered from the video capture of facial expression and related neural activity was measured using wearable and portable neuroimaging systems: functional near infrared spectroscopy (fNIRS), electroencephalography (EEG) to asses both hemodynamic and electrophysiological responses. The methodology involves the simultaneous detection and comparison of various affective expressions by multi-modalities and classification of spatio-temporal data with neural signature traits. The experimental results show strong correlation between the spontaneous facial affective expressions and the affective states related brain activity. We propose a multimodal approach to jointly evaluate fNIRS signals and EEG signals for affective state detection. Results indicate that proposed method with fNIRS+EEG improves performance over fNIRS or EEG only approaches. These findings encourage further studies of the joint utilization of video and brain signals for face perception and brain-computer interface (BCI) applications.