In Situ Analysis of Electrodermal Activity from Students Learning from Large Language Models Versus Curated Texts

Background: this paper reports an investigation into the cognitive and emotional states of adolescents while learning from an LLM. It seeks to address a relative dearth in empirical evidence which might otherwise facilitate informed decisions being made by curriculum designers, school leaders and policy makers regarding the use of Generative AI, amidst the wider discourse about the effectiveness of AI in teaching and learning. Methods: in this paper, we analyze electrodermal activity (EDA) in the context of students’ scholastic engagement using LLMs in comparison to curated texts. In our 27-min-long experiment, we recorded the EDA of participants learning from both learning methods, for 8 min each. A quiz was also conducted to assess the effectiveness of the learning method. We collected 23 samples of EDA from the experiment, and 42 samples of quiz results. Results: we have found that learning with an LLM results in greater Skin Conductance Response (p = 0.09404), which is linked to more positive emotional valence, and lower Skin Conductance Level (p = 0.09473), which is linked to lower cognitive load, compared to curated texts. We also discovered that learning with an LLM correlates to a higher quiz result (p = 0.02053). While this suggests that learning and absorbing information with an LLM could be more effective than curated texts, results from self-reported data indicate that there are few perceived differences between the effectiveness of LLM and curated texts. Conclusions: this exploratory and preliminary study revealed empirical insights between LLM usage and learning effectiveness in situ via physiological indicators, in contrast to prior work that has adopted post hoc frames over the medium- to long-term.