ABSTRACT Background As researchers rush to investigate the potential of AI tools like ChatGPT to enhance learning, well‐documented pitfalls threaten the validity of this emerging research. Issues of media comparison research, where the confounding of instructional methods and technological affordances is unrecognised, may render effects uninterpretable. Objectives Using a recent meta‐analysis by Deng et al. ( Computers & Education , 227, 105224) as an example, we revisit key insights from the media/methods debate to highlight recurring conceptual challenges in ChatGPT efficacy studies. Methods This conceptual article contrasts nascent ChatGPT research with the more established literature on Intelligent Tutoring Systems to identify three non‐negotiable considerations for interpretable effects: (1) descriptions of the precise nature of the experimental treatment and (2) the activities of the control group, as well as (3) outcome measures as valid indicators of learning. To provide some initial evidence, we audited a subset of primary experiments included in Deng et al.'s meta‐analysis, demonstrating that only a small minority of studies satisfied all three non‐negotiable considerations. Results and Conclusions Loosely defined treatments, mismatched or opaque controls, and outcome measures with unclear links to durable learning obscure causal claims of this emerging literature. Observed gains cannot, at this time, be confidently attributed to ChatGPT, and meta‐analytics effect sizes may over‐ or understate its benefits. Progress, we argue, will require rigorous designs, transparent reporting, and a critical stance toward “fast science.”
Weidlich et al. (Tue,) studied this question.