Answers are grounded in this paper and cite their sources.
Research on enhancing error message presentation is of great interest to teachers and developers alike because improving Integrated Development Environments (IDEs) increases early student retention and efficiency at all levels with more effective developing tools. This study aims to compare GPT-4 and novice programmer accuracy in fixing errors to assess the viability of Large Language Models as an error message enhancement tool. First, a random sample of 100,000 sessions from all users of BlueJ 5, an IDE for novice programmers, was analyzed to determine the time it took programmers to resolve coding errors. Subsequently, for each of the five most common errors, GPT-4 was given 20 randomly-selected snippets of code from Blackbox mini, a curated subset of Blackbox with source code attached, and prompted to explain and fix the errors. This study replicated prior research that proposed a Zipf-Mandelbrot Distribution of error message frequency; the five most common errors comprised 45% of all error messages. In comparing GPT-4 and novices, it was found that humans fix code at higher rates, but GPT-4 provided completely correct explanations for error messages 96% of the time. This study concludes that GPT-4 functions best as a tool to explain error messages in an interactive format, rather than as a tool to produce correct code on its own. In conclusion, GPT-4 would be best utilized to enhance the classroom experience as a chat assistant to reduce time spent on syntactical errors, leading to improved productivity and better novice retention.
No takes yet. Share an insight, caveat, or question.
Brij Howard-Sarin (2024) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: