Scoring consistency of teachers and artificial intelligence algorithms in written expression assessment: a comparative quantitative study

Written expression assessment at the secondary school level involves complex interactions among text genre, evaluator type, and student characteristics. The present study examined the degree of agreement, divergence, and patterning between scores produced by four large language model (LLM)-based algorithms, ChatGPT, Grok, Minimax, and Gemini, and those assigned by human teachers when rating texts composed by secondary school students. A total of 122 students (63 female, 59 male), aged approximately 11–14 years and enrolled in grades 6, 7, and 8 at a public secondary school in Turkey, participated in the study and composed their texts in Turkish. Each student produced one text in a randomly assigned genre: folktale, poetry, or biography. All raters, both human teachers and AI algorithms, evaluated the texts using an identical five-dimensional analytic rubric. Inter-rater agreement was quantified via Pearson correlations and intraclass correlation coefficients (ICC), while score differences across evaluator type, text genre, and gender were examined using mixed-design ANOVA with Bonferroni-adjusted paired-samples t-tests for pairwise comparisons and independent-samples t-tests. ChatGPT demonstrated the highest alignment with teacher scores ( r = .78, ICC = 0.78). In contrast, Minimax showed the weakest correspondence ( r = .71). Text genre did not exert a statistically significant main effect on scores (F(2, 119) = 0.81, p = .445, partial η 2 = 0.01); however, a statistically significant evaluator-by-genre interaction was detected (F(8, 476) = 3.77, p < .001, partial η 2 = 0.06), such that human teachers scored poetry markedly lower than the other genres. In contrast, AI systems did not show this pattern. Gender-related score differences were observed exclusively in teacher evaluations (d = 0.50), whereas no statistically significant gender differences were detected in any of the four AI algorithms’ scores (all p ≥ .198). These findings indicate that LLM-based algorithms hold promise as formative feedback tools particularly ChatGPT, whose scores aligned most closely with teacher judgments across dimensions and genres, making it best suited to generating rapid, low-stakes draft feedback that students and teachers can use to guide revision while also revealing algorithm-specific limitations, most notably Minimax’s weaker alignment with teacher scores, that preclude direct substitution for human judgment in high-stakes writing assessment.