Automated Benchmarking Infrastructure: Moving Toward Robust Investigations of Gen AI in Computing Education

D uring a 12-week term in 2024, an average of 32 new large language models (LLMs) were released [4].The most powerful LLMs can generate code explanations, improve error messages, generate code itself, and even simulate students [9,10,11,12].While many LLMs are relatively easy to access and use for computing educators through the Web, consider an educator who wants to see how easy it would be for a student to use the latest version of ChatGPT or Gemini to answer their exam questions.They would have to meticulously structure their prompts to pass into the AI tools and then arduously read through the essay-length, AI-generated outputs for each question to determine correctness.And then they would repeat the process for Gemini.This process is repetitive and tedious, especially when considering that 30+ other new LLMs will become available throughout the term.The educator simply does not have the capacity to keep up.In the spirit of computer science (CS) and automating repetitive tasks, we propose using existing AI benchmarking infrastructure to support more efficient evaluations and reliable comparisons of rapidly evolving LLMs.