AI-assisted, instructor-supervised grading and feedback in higher education: Design and evaluation of an end-to-end pipeline

This paper presents the design, deployment and evaluation of a GPT-4o-based grading and feedback pipeline for a Master’s-level Information Systems Project Management course, developed following the Design Science Research Methodology. The artefact orchestrates submission collection (Google Forms), rubric-based scoring with rubric-linked qualitative feedback (OpenAI GPT-4o via a Cloud Function) and analytics storage (BigQuery), across six consecutive case-study assignments. The AI grader supplemented rather than replaced instructor grading: official grades remained the instructor’s, while AI grades and feedback served as a second-opinion signal and additional formative feedback. The evaluation followed a pre-specified analysis plan combining a quantitative comparison of AI and instructor grades, a voluntary student survey and an automated assessment of feedback quality on the full corpus. Across 362 submissions from 66 students, the AI reproduced the instructor’s mark within the pre-specified tolerance of 0.5 points on a five-point scale in 83% of cases, with a mean absolute error (MAE) of 0.31 points and a small systematic bias of −0.16 points (95% CI [ − 0.19 , − 0.12 ] ). Chance-corrected agreement, measured by the intraclass correlation coefficient (ICC), was moderate at best: ICC(2,1) = 0.49 (95% CI [ 0.33,0.61 ] ), spanning the poor-to-moderate range. Among the 112 survey responses (31% response rate), about 80% rated the feedback useful and 77% accurate. The pipeline operated at EUR 0.55 per student per assignment and delivered feedback within 24 h. The findings support using LLMs as scalable assistants for formative assessment, as a supplement to rather than a substitute for instructor evaluation.