Personalized Learning Squared (PLUS): Implementation of Large Language Model-Facilitated Assessment of Tutors’ Performance in Reacting to Students Making Errors

Personalized Learning Squared (PLUS) is an integrated tutoring platform led by Carnegie Mellon University in collaboration with Carnegie Learning, Inc and Stanford University. PLUS employs a system of hybrid human-AI tutoring, which combines both AI-assisted tutoring software and human tutoring to provide high-impact math tutoring to economically disadvantaged and under-resourced middle school students with a focus on enhancing the skills of novice, inexperienced tutors. Tutors require robust training to effectively assist students, and currently, there is a lack of trained tutors. Inexperienced and novice tutors lack specific scenario-based knowledge related to more nuanced skills and competencies in tutoring. Understanding this, PLUS has over 20 scenario-based lessons on specific competencies including Responding to Negative Self-Talk, Determining What Students Know, and Ensuring Conceptual Understanding, with the aim of enhancing the skills of novice tutors. In this demo, we showcase how large language models (LLMs) can be used to assess particular tutor criteria related to the Reacting to Student Errors lesson with three crucial perspectives: 1). the perspective of a tutor-supervisor to assess tutor competencies; 2). the perspective of a tutor engaging with the lesson and the AI-generated constructive questions; 3). the perspective of an AI-evaluator employing LLMs to compare human and AI assessment of tutors. This PLUS research-backed approach can extend to other competencies to enhance the future of education.