Multimodal Emotion Recognition Based on Ensemble Convolutional Neural Network

In recent years, emotional recognition based on Electrophysiological (EEG) signals has become more and more popular. But the researchers ignored the fact that peripheral physiological signals can also reflect changes in mood. We propose an Ensemble Convolutional Neural Network (ECNN) model, which is used to automatically mine the correlation between multi-channel EEG signals and peripheral physiological signals in order to improve the emotion recognition accuracy. First, we design five convolution networks and use global average pooling (GAP) layers instead of fully connected layers; and then the plurality voting strategy is adopted to establish the ensemble model; eventually this model divides emotions into four categories. Based on the simulations on DEAP dataset, the experimental results demonstrate the superiority of the ECNN compared with other methods.