Comparative evaluation demonstrates GPT-4 matches human examiners in grading open-text macroeconomics questions, suggesting potential for automated grading systems in university education.
Abstract Under what circumstances can GPT-4 be relied upon as a safe and efficient decision-maker? To address this question we explore the performance of GPT-4 as an examiner of answers to open text questions. We formulate questions and sample solutions in the subject area of macroeconomics and collect answers from cohorts of students. Subsequently, we conduct a fair competition between GPT-4 and human experts, employing their expertise to assess the quality of the answers. We observe that the substitution of GPT-4 for a human examiner does not decrease inter-rater reliability. We run checks on potential biases (whether GPT-4 prefers AI-generated or lengthy answers). We find no consistent evidence of such biases. Our findings are robust to tilting the competition to one side's advantage, by using inferior or advanced prompting strategies. These findings suggest a broad range of potential applications for GPT-4 in university education. Overall, our study cautiously supports the utilization of GPT-4 as an assistant for automated grading systems.
No takes yet. Share an insight, caveat, or question.
Zubaer et al. (2024) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: