Aligning artificial intelligence and human judgment in IELTS speaking assessment to develop a hybrid pedagogical framework for test preparation

This study investigates the alignment between human judgment and artificial intelligence (AI) in IELTS speaking assessment to inform effective pedagogical practices. Through a comparative analysis of 30 Indonesian EFL learners’ performances evaluated by two experienced human raters and three AI systems (ChatGPT, Otter AI, Claude AI), we identified systematic and significant differences that varied considerably across individual AI systems. When AI scores were averaged, human evaluators assigned higher scores than the automated systems overall (mean difference = 0.522, p < .001), with a moderate positive correlation ( r = .497, p = .005). However, disaggregating the AI systems revealed that this overall pattern was driven primarily by two of the three tools: ChatGPT scores did not differ significantly from one of the two human raters, while Otter AI and, in particular, Claude AI diverged substantially from both human raters. Beyond quantifying these differences, this paper explores their profound implications for language teaching and learning. The findings reveal AI’s superior consistency in measuring technical accuracy (e.g., pronunciation, grammar) and human raters’ strength in assessing communicative effectiveness. We argue that this complementarity presents a critical opportunity rather than a problem. The research concludes by proposing a practical hybrid pedagogical framework where AI is leveraged for low-stakes practice, instantaneous feedback, and freeing up teacher time for high-value instruction on pragmatic and interactive speaking skills. This model positions AI not as a replacement for human judgment but as a transformative tool to enhance teaching efficiency and student learning outcomes in preparation for high-stakes speaking assessment.