Answers are grounded in this paper and cite their sources.
Formative assessment supports student learning when feedback is timely and consistent, but its delivery at scale is constrained by instructor workload, student-data privacy, and uneven access to cloud-based AI services across higher education institutions. These conditions affect which students receive regular formative feedback and under what supervision their work is evaluated. This study examined whether smaller language models (≤32B parameters), which are more likely to be deployable on institutional infrastructure, can support human evaluators in formative programming assessment under a human-in-the-loop configuration and how they compare with larger models (≥70B). Eleven models (six smaller, five larger) were compared against the consensus of four trained instructors across 308 response-level evaluations from three assignments in a Data Structures and Algorithms course, covering text and code questions. The ±5 and ± 10 point equivalence bounds were calibrated against the pairwise disagreement observed between the four evaluators, providing course-specific anchors. Inter-rater reliability, based on Intraclass Correlation Coefficient (ICC), was high for single evaluators, ICC(2,1) = 0.901, and the four-evaluator consensus mean, ICC(2,k) = 0.973. On text, two smaller models reached strict ±5 point equivalence with the human consensus, against one of five larger models. On code, three smaller and one larger model reached moderate ±10 point equivalence. Across question types, smaller models showed higher instructor review rates than larger models. These findings indicate that specific smaller models can match larger models in formative programming assessment under instructor review, with open-source options available as a viable on-premise alternative for institutions of different resource levels.
No takes yet. Share an insight, caveat, or question.
Mendonça et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: