We experimented with ChatGPT's ability to write programs in a CS1 class, and the ability of a popular tool to auto-detect ChatGPT-written programs. We found ChatGPT was proficient at generating correct programs from a mere copy-paste of the English programming assignment specifications. However, running ChatGPT for 10 programming assignments and acting as 20 different students, and using zyBook's APEX beta tool for academic integrity, we found: (1) ChatGPT-generated programs tend to use a programming style departing from the style taught in the textbook or by the instructor, and these "style anomalies" were automatically detected. (2) Although ChatGPT may for the same assignment generate a few different program solutions for different students, ChatGPT often generates highly-similar programs for different students, so if enough students in a class (e.g., 5 or more) use ChatGPT, their programs will likely be flagged by a similarity checker. (3) If students are required to do all programming in the autograder's IDE, then a student using ChatGPT ends up showing very little time relative to classmates, which is automatically flagged. (4) Manually, we observed that if a student consistently uses ChatGPT to submit programs, the programming style may vary across programs, something normal students don't do; automation of style inconsistency detection was recently added to APEX. In short, while there will no doubt be an arms race between AI-generated programs and automatic detection of AI-generated programs, currently students using ChatGPT for multiple CS1 programs can be detected by automated tools such as zyBooks' APEX.
No takes yet. Share an insight, caveat, or question.
Pang et al. (2024) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: