What question did this study set out to answer?

This study aims to evaluate how well rubric-guided large language models score compared to human grading in higher education.

June 14, 2026Applied SciencesOpen Access

Evaluating the Reliability and Agreement of Rubric-Guided LLM Scoring Versus Human Grading Across Three University Courses

View Full Paper

Ask AI

Bookmark

This study aims to evaluate how well rubric-guided large language models score compared to human grading in higher education.
Analyzed 930 student responses from three courses using a five-criterion rubric.
Scores were compared between two human raters and a large language model (ChatGPT).
Utilized multiple statistical methods including ICC, Pearson correlations, and Bland–Altman analysis for agreement assessment.
Human consensus showed strong agreement (ICC = 0.819).
AI–H1 agreement had an ICC of 0.700, while AI–H2 scored ICC = 0.767 and AI–HC agreement was ICC = 0.763.
Calibration improved Total-score ICC from 0.774 to 0.782, with MAE reduction from 1.624 to 1.215.

No takes yet. Share an insight, caveat, or question.

Randomized trial assesses agreement of rubric-guided LLM scoring with human grading, suggesting supportive AI co-grading.

Kim et al. (2026) studied this question.

View Full Paper

Ask AI

Bookmark