Students’ emotion extraction and visualization for engagement detection in online learning

Online learning is growing in various forms, including full-online, hybrid, hy-flex, blended, synchronous, and asynchronous. Assessing students’ engagement without having real contact between teachers and students is becoming a challenge for the teachers. Therefore, this paper focuses on analyzing online lecture videos to detect students’ engagement without relying on learning management systems produced data. In this regard, an intelligent application for teachers is developed to understand students’ emotions and detect students’ engagement levels while a lecture is in progress. Real-time and offline lecture videos are analyzed using computer vision-based methods to extract students’ emotions, as emotions play essential roles in the learning process. Six types of basic emotions, namely ‘angry’, ‘disgust’, ‘fear’, ‘happy’, ‘sad’, ‘surprise’, ‘neutral’ are extracted using a pre-trained Convolutional Neural Network (CNN), and those are used to detect ‘Highly-engaged’, ‘Engaged’, and ‘Disengaged’ students in a virtual classroom. This educational application is tested on a 28-second-long lecture video taken from YouTube consisting of 11 students. The results are visualized for engagement detection using visualization methods. Furthermore, this intelligent application, in real-time, is capable of recognizing multiple faces when multiple students share a single camera. Nonetheless, this educational application could be used for supporting collaborative learning, problem-based learning, and emotion-based grouping.