There is increased interest in one aspect of this, namely the identification of emotion as an important field in affective computing, in which the ability of intelligent systems to analyze human emotions has been introduced and the efficiency of human–machine interactions has been improved. Facial expression recognition (FER) is the only modality used in traditional approaches to emotion recognition and consequently these approaches are less robust because of inter-personal and inter-environmental variability. In this paper, the recent advances in multimodal emotion recognition (MER) systems that involve combining the facial expression recognition through deep learning approaches with physiological signal-based emotion recognition (PSER) are reviewed. An overview of the reviewed works shows that a conceptual multimodal framework can be presented that consists of data acquisition, data preprocessing, feature extraction, multimodal fusion, and classification stages. Previous studies have extensively used Convolutional Neural Networks (CNNs) for facial feature extraction, while physiological signals like electrodermal activity (EDA) and heart rate variability (HRV) have been traditionally analyzed using statistical and frequency domain approaches. Emotional pattern learning across the various domains and fusion of heterogeneous multimodal representations can be achieved by feature-level fusion strategies, as reported in the literature. Recent studies in the literature have shown that temporal modeling approaches have proven to be more stable when facing dynamic environmental conditions. Architectures reviewed are typically modular and designed to be scalable and adaptable for future real-life implementation scenarios. The aim of the present work is primarily to review the latest emotion recognition systems based on multimodal approach and to present a conceptual framework based on the literature. Future research directions are suggested to be experimental implementation and quantitative validation.
Shinde et al. (Thu,) studied this question.