A comparison of students' emotional self-reports with automated facial emotion recognition in a reading situation

This study investigated the measurement of students' emotional states during a common learning activity, digital reading of factual texts. The objective was to compare emotional self-reports with automated facial emotion recognition. The latter promises non-intrusive measurements of emotions, which could inform adaptive learning systems. We used an established facial emotion recognition software trained on experts' ratings of facial expressions (FaceReader). For basic emotions, previous studies have reported high agreement of the software with human raters. However, little evidence exists a) on its performance for the epistemic emotions of interest and boredom, b) on its agreement with self-reports, and c) in naturalistic reading situations. We compared the facial expression-based recognition of interest, boredom, and valence of affect to students' self-reports of those emotional states. Analyses of webcam recordings of 103 students revealed no relationship between facial emotion recognition and self-reports. Due to the low agreement of the facial emotion recognition software with self-reports, it remains unclear what the facial expression-based recognition of interest, boredom, and valence actually implies. We advise to wait for more comprehensive evidence a) on the agreement of facial emotion recognition software with self-reports or b) on its predictive validity for learning before applying it in educational practice (e.g., in adaptive learning systems).