Privacy-preserving MoE sparse transformer for trimodal emotion recognition in classroom environments

Emotion recognition in classrooms plays a crucial role in enhancing human-computer interactions, especially in educational environments. This paper proposes a novel MoE-Sparse Transformer architecture for classroom emotion recognition that integrates visual, auditory, and textual data. The model leverages the Mixture-of-Experts (MoE) mechanism and sparse attention to efficiently process multimodal streams, improving computational efficiency while maintaining high accuracy. Additionally, it incorporates Identity Irreversible Desensitization (IID) to decouple identity-related features from emotional signals, ensuring privacy protection. We evaluate our model on three multimodal datasets (SAMSEMO, EmotionTalk, and IEMOCAP), achieving an average Weighted Average F1-score (WA-F1) of 78.5% and Unweighted Average Recall (UAR) of 74.8% across these datasets. Our model outperforms existing methods, such as TFN and HyFusER, by 4–6% in WA-F1. While incorporating IID slightly reduces performance by 0.4−0.6%, it effectively ensures privacy with an identity leakage rate of 53.8%, compared to the 86.3% leakage in models without IID. This approach provides a robust solution for emotion recognition in both traditional and remote classroom settings, balancing performance and privacy protection.