Impact of guided AI Interaction on secondary students’ physics learning outcomes: evidence from a cluster-randomized longitudinal design and multilevel model

This study aimed to examine the impact of AI-supported learning environments on secondary students’ physics learning outcomes by comparing a guided ChatGPT interaction design with open chatbot use. A facilitator-matched cluster-randomized longitudinal online trial was conducted among 604 Jordanian Grade 11 and Grade 12 students participating in a six-week Arabic-language physics learning program between February and April 2026. Participants were assigned to 24 supervised virtual cohorts, with 12 cohorts receiving a structured guided-prompting approach and 12 cohorts engaging in open ChatGPT use. Both conditions used the same ChatGPT model, learning materials, instructional tasks, session duration, and facilitator support. Because participants were nested within virtual cohorts and facilitator-matched pairs, multilevel mixed-effects models were applied to examine changes in conceptual understanding while accounting for baseline performance, cohort clustering, and facilitator effects. The analysis showed that guided AI-supported learning significantly improved immediate conceptual understanding compared with open chatbot use after controlling for baseline differences and design factors (adjusted difference = 3.07 points, 95% CI [1.43, 4.72], p = .002, Hedges’ g = 0.48). Guided interaction also produced stronger immediate transfer (adjusted difference = 2.54, p < .001, g = 0.53) and delayed retention (adjusted difference = 3.01, p = .001, g = 0.40). Transcript analyzes further indicated that guided learning environments promoted more evidence seeking, critical examination, verification, reflection, and revision, while reducing direct-answer seeking. These findings show that the educational value of generative AI depends not only on access to a capable system but also on the instructional structures that shape what learners do with its output. The results support designing AI-supported physics learning environments that preserve students’ cognitive responsibility and foster transferable learning.