Exploring the use of a customised chatbot on time-limited Python tasks in undergraduate physics

Generative artificial intelligence (GenAI) is transforming how novices learn to programme, yet evidence of effective and equitable use in physics education remains limited. We report an exploratory controlled study ( N = 27) in which undergraduates completed a 45 min, 16-item Python task under one of three support conditions: Internet (standard web resources), GPT (ChatGPT-3.5), or Custom-GPT (a hint-based variant configured to scaffold reasoning while avoiding runnable code). The primary outcome was percentage score; secondary outcomes were items attempted and accuracy on attempted items, complemented by a brief post-task survey, follow-up interviews, and chatbot log analysis. Mean scores were 67 ± 4%, 74 ± 5%, and 79 ± 3% for Internet , GPT , and Custom-GPT respectively. No pairwise comparison reached statistical significance after correction for multiple comparisons; however, effect size analysis revealed a large effect favouring Custom-GPT over Internet (Cliff’s δ = 0.51[−0.01, 0.91], though the confidence interval narrowly includes zero). Students attempted more questions with GPT and Custom-GPT (both 15/16) than with Internet (13/16), while accuracy on attempted items was comparable across groups. Descriptively, a novice-support pattern emerged: among students with <1 year of Python, Custom-GPT showed the largest advantage over Internet (21 percentage points), though subgroup sizes ( n = 3) preclude inferential claims. Survey and interview data highlighted speed, targeted guidance, and confidence-checking as benefits, along with concerns about shallow learning when solutions were too direct. Given the small sample, findings are exploratory and hypothesis-generating. Overall, in a timed setting, GenAI supported students primarily by reducing search and debugging friction, with the largest gains from Custom-GPT .