Exploring LLM autoscoring reliability in large-scale writing assessments using generalizability theory

This study investigates the accuracy and reliability of large language models (LLMs) in scoring writing tasks from the Advanced Placement (AP) Chinese Language and Culture Exam. Using generalizability theory, the research evaluates and compares score consistency (reliability) and alignment with human raters (accuracy) across two types of AP Chinese free-response writing tasks: story narration and email response. These essays were independently scored by two trained human raters and seven artificial intelligence (AI) raters. Each essay received four scores: one holistic score and three analytic scores corresponding to the domains of task completion, delivery, and language use. Results indicate that although human raters produced more reliable scores overall, LLMs demonstrated reasonable consistency under certain conditions, particularly for story narration tasks. Composite scoring that incorporates both human and AI raters enhanced score dependability and human–AI agreement, which supports that hybrid scoring models improve both reliability and accuracy in large-scale writing assessments.