Evaluating ChatGPT and GPT-4 for Visual Programming

Generative AI has the potential to drastically improve the landscape of computing education by automatically generating personalized feedback and content. In particular, this potential lies in the advanced capabilities of state-of-the-art deep generative and large language models such as OpenAI’s Codex [7], ChatGPT [11], and GPT-4 [12]. In our work, we seek to investigate the capabilities of these models in visual programming domains popularly used for K-8 programming education, including domains like Scratch [17], Hour of Code: Maze Challenge by Code.org [4, 5], and Karel [13].