A comprehensive deep learning framework for real time emotion detection in online learning using hybrid models

Abstract This paper introduces an advanced Facial Emotion Recognition (FER) system that integrates ResNet-50, the Convolutional Block Attention Module (CBAM), 3D Convolutional Neural Networks (3D CNN), and Ant Colony and Genetic Algorithm-based Target Optimization (AGTO). The proposed model is meticulously evaluated to identify the most effective predictive classification model for real-time engagement detection. By leveraging facial emotions, this deep learning-based system monitors the real-time engagement of online learners and is tested on multiple FER datasets, achieving notable accuracies: 95.57% on FER2013, 97.29% on CK+, 98.35% on KDEF, and 98.09% on a proprietary dataset, demonstrating significant improvements over existing approaches. Comparative analyses against state-of-the-art models highlight the importance of these findings for educational institutions. This approach enhances emotion recognition accuracy, refines feature relevance, captures temporal dynamics, enables real-time monitoring, and ensures robustness and adaptability in online learning environments. The integrated capabilities of ResNet-50, CBAM, 3D CNN, and AGTO contribute uniquely to capturing dynamic facial expression changes, enabling precise interpretation of students’ emotions and engagement levels. The proposed system achieves a facial emotion classification accuracy of 97.3% in real-time learning scenarios, surpassing current methodologies.