Algorithmic Rater Drift in Automated Essay Scoring: A Many-Facet Rasch Analysis of Successive LLM Versions

Although large language models are increasingly used for automated scoring, the psychometric stability of their scoring across version updates remains underexplored. This study compared two successive ChatGPT versions (GPT-5.1 and GPT-5.2) on persuasive essays in agreement, severity, leniency, and bias. A stratified sample of 99 ASAP 2.0 essays was scored three times by each model with a holistic rubric and analyzed using agreement indices, Bland–Altman analysis, and Many-Facet Rasch Measurement (MFRM). Inter-version correlation was high (r = .771) and adjacent agreement reached 99.0%, yet MFRM revealed a 4.26-logit severity–leniency difference. Against human scores, GPT-5.1 was nearly unbiased (−0.04; 60.6% exact agreement), whereas GPT-5.2 was systematically lenient (+0.33; 47.5%). Raising the reasoning-effort setting from none to medium made both versions more lenient and less stable. Inter-version consistency does not guarantee agreement with human scores; version updates can produce algorithmic rater drift. Version locking and criterion-based validation are recommended.