Disengaged Conversational Behaviors and Learning in Student-AI Dialogues for Mathematical Problem Solving

As the conversational AI agents become integrated into digital learning environments, students’ interaction with these systems can show conversational disengagement, including mocking, misleading, or otherwise diverting the agents. Although task-based disengagement such as gaming the system and off-task actions are well documented, the nature of the consequences of this conversational disengagement in student-AI dialogue remains largely unexplored. This study examined how different types of conversational disengagement relate to middle-school students’ mathematical learning with an AI-powered teachable agent embedded in a large-scale platform. We coded 5,522 utterances from 246 students into six conversational disengagement categories (False Answering, Mocking, Ghosting, Mirroring, Withholding, and Random/Noisy) versus engagement. Regression analyses showed that some disengagements (False Answering and Mocking) were positively associated with post-test scores after controlling for pre-test performance, although these associations did not replicate in a sensitivity analysis restricted to human-annotated utterances. To capture process-level dynamics, we modeled how conversational disengagement ends using a competing-risks approach and separately clustered students based on their turn-by-turn transition probabilities. Most conversational disengagement tended to end within the first few turns, whereas Withholding showed a strong tendency to continue. Students in the highest-performing cluster combined a moderate level of conversational disengagement with strong recovery pathways, whereas the lowest-performing cluster exhibited persistent self-loops without recovery. Together, these findings suggest that some apparent disengagement in conversation with AI may occur within interaction trajectories that subsequently return to productive engagement.