Traditional educational robots mostly rely on single-mode emotion recognition, so it is difficult to fully capture students' emotional state. Emotional factors have a significant impact on English learners' oral expression and learning motivation, so accurate emotional recognition is very important. In this study, multimodal data such as speech, text semantics, physiological signals and body movements are collected, and after preprocessing, a three-stage Cross-Modal Fusion Network (CMFN) architecture is constructed, including monomodal feature coding, adaptive attention fusion and emotion classification output. The model realizes accurate emotion recognition by dynamically assigning modal weights. The experiment recruited 60 English learners and collected multimodal data for testing. The results show that CMFN model is significantly superior to unimodal and bimodal models in terms of arousal, valence MAE and discrete emotion accuracy, with an accuracy rate of 87.3%, and still maintains high robustness in different noise scenes. This model can provide data support for English educational robots, help adjust individualized teaching strategies and improve teaching effect.
Yun Feng (Wed,) studied this question.