A Multimodal Affective Computing Framework for Real‐Time Student Engagement Assessment in IoT ‐Enabled English Classrooms: An Edge‐Cloud Collaborative Approach

ABSTRACT In the English teaching environment driven by the Internet of Things, real‐time assessment of students' emotional states based on video data is particularly crucial for understanding student engagement and improving teaching quality. Existing server‐based deep networks are limited by long‐distance video data transmission, which seriously restricts the real‐time performance of sentiment analysis. Moreover, simple video data cannot guarantee robustness in complex scenes. To address these issues, this paper proposes a multimodal fusion emotion recognition framework based on the edge‐cloud collaboration mechanism. Firstly, on the edge node, we exploit two complementary modalities of data: video sequences and facial landmark sequences, and design a lightweight dual‐stream neural network based on the 3D MobileNetV3 and graph convolutional network to efficiently extract multimodal features. On the server, we adopt the Transformer‐based cross fusion mechanism to implement multimodal fusion and emotion evaluation. The edge side is responsible for real‐time preprocessing and primary feature extraction. In our proposed framework, the server is responsible for aggregating feature data from multiple edge nodes. The experimental results indicate that the proposed framework can achieve high‐precision student engagement assessment with low latency.