Abstract Metacognitive reflection is a crucial transversal skill, especially in an era where generative AI transforms how we teach and learn. As well as being a driver of the need to develop metacognitive reflection, generative AI is also a tool that can be used to enhance metacognitive reflection, such as chatbots that act as coaches to guide students in metacognitive reflective practice. In this study, we examined the potential of LLM-powered chatbots to promote metacognitive reflection across three distinct educational contexts. Our results show that the chatbot successfully constructed a metacognitive dialogue and delivered relevant, evidence-based recommendations. However, student engagement levels were generally low, with limited active participation observed across all studies. Notably, metacognitive self-regulation, and other individual differences, did not consistently predict engagement levels, suggesting that learners with higher reported self-regulation were not inherently more likely to use the tool. We also found no evidence that metacognitive engagement levels led to improved learning outcomes. However, these findings must be interpreted with caution, as engagement levels may be a limited metric for capturing how students benefit from chatbot-assisted reflection. We conclude by raising key design questions around how to develop chatbot systems that not only deliver metacognitive content and feedback but also encourage active student participation. While system prompts can help LLMs maintain focus on metacognitive reflection, hybrid designs that add an additional layer of scripting or multi-agent systems may be necessary to support an active learner role and ensure that important metacognitive checkpoints are met by the learner.
Uittenhove et al. (Tue,) studied this question.