Using ChatGPT with Novice Arduino Programmers: Effects on Performance, Interest, Self-Efficacy, and Programming Ability

A posttest-only control group experimental design compared novice Arduino programmers who developed their own programs (self-programming group, n =17) with novice Arduino programmers who used ChatGPT 3.5 to write their programs (ChatGPT-programming group, n = 16) on the dependent variables of programming scores, interest in Arduino programming, Arduino programming self-efficacy, Arduino programming posttest scores, and types of programming errors. Students were undergraduates in an introductory agricultural systems technology course in Fall 2023. The results indicated no significant (p < .10) differences between groups for programming rubric scores (p = .50) or interest in Arduino programming (p = .50). There were significant differences for Arduino programming self-efficacy, (p = .03, Cohen's d = 0.75) and Arduino posttest scores, (p = .03, Cohen's d = 0.76); students in the self-programming group scored significantly higher on both measures. Analysis of students' errors indicated the ChatGPT group made significantly (p < .01) more program punctuation errors. These results indicated novice students writing their own programs developed greater Arduino programming self-efficacy and programming ability than novice students using ChatGPT. Nevertheless, ChatGPT may still play an important role in assisting novices to write microcontroller programs.