Evaluating and Mitigating Second-Language Bias in LLM-Based Writing Feedback with Category-Resolved Counterfactual Invariance

Scorers built on large language models (LLMs) increasingly drive analytic feedback for English language learners (ELLs). Learner errors should lower the scores for conventions or grammar, but they can also spill over into dimensions that the rubric defines independently of them, such as cohesion or vocabulary, and redirect the feedback a learner receives. We audit an LLM-based analytic feedback model on 1,200 ELLIPSE essays with category-resolved counterfactuals, namely spelling, grammatical-morphology and preposition errors whose rates are re-estimated from JFLEG and TOEFL-Spell, and with authentic misspellings corrected in the students’ own essays. Added misspellings lower the predicted cohesion, syntax and vocabulary scores by about half as much as conventions and change the rounded scores of about a quarter of the learners. We propose CR-CAIR, which extends construct-aligned invariance regularization with a rubric-derived category–trait construct map and a sensitivity-anchoring term while keeping a closed-form solution. Over five splits, CR-CAIR reduces spill-over on construct-irrelevant cells by 86% and the corresponding flip rate from 12.2% to 4.2%, keeps the legitimate sensitivity of conventions, grammar and phraseology, nearly halves misdirected feedback priorities and transfers to authentic misspellings (−79% spill-over), with no significant change in agreement (QWK 0.649 vs. 0.654). Task-agnostic debiasing removes legitimate sensitivity, and trait-level invariance increases misdirected feedback.