Ensuring reliable essay scoring is challenging when rater severity and bias vary by linguistic background. This study investigates whether a GPT-based Automated Essay Scoring (AES) system can complement human raters in second-language (L2) writing assessment. Using a Many-Facet Rasch Measurement (MFRM) model, this study analyzed ratings from 20 human raters—10 native English-speaking (NES) and 10 non-native English-speaking (NNES)—and GPT across four analytic criteria ( Content , Organization , Grammar , and Vocabulary ) on 181 university L2 essays. The results indicate that raters demonstrated variability and potential rater bias; GPT exhibited high internal consistency but range compression at the upper end compared to humans. These findings offer insights into how GPT can supplement or complement traditional rater-based methods, potentially alleviating the time-consuming and subjective aspects of human scoring. Nonetheless, concerns remain regarding its sensitivity to subjective writing features such as creativity, tone, and nuanced lexical use. This study contributes to the growing understanding of large language models in educational contexts and highlights the need for further refinement and validation of AES systems.
Wu et al. (Sun,) studied this question.