Methodological audit assesses AI reporting in education research, suggesting improvements for accountability.
We audit how the use of artificial intelligence is reported in recent education research. From a harvest of 29 848 arXiv preprints and 543 articles in six education and education-technology journals (2025-2026), we coded 220 papers (127 arXiv + 93 journal) against a 19-item checklist adapted from the TRIPOD-LLM reporting guideline, plus descriptive and outcome items. Coding was performed by open-weight large language models (served through Ollama) from titles and abstracts, conservatively (an item not stated is coded 0); we report cross-model agreement (mean Cohen's κ = 0.53, raw agreement 87%) in place of inter-human reliability, and disclose the AI-coded method in full. Overall reporting compliance is low: the median paper reports 32% of the checklist items, and the lowest-compliance items are the cross-cutting accountability signals - funding and conflicts of interest, missing-data handling, calibration/fairness, compute and cost, and the human-in-the-loop protocol (each ≤ 7%). Reporting quantity does not differ between arXiv preprints and journal articles in the unadjusted comparison (equal medians; unadjusted odds ratio ≈ 1); what differs is composition - preprints document the model machinery while journal articles document the study context, and neither documents accountability. A modest journal advantage in quantity emerges only after adjusting for study design. Empirical design is the dominant predictor of how many items a paper reports. A within-paper preprint-vs-published comparison was planned but could not be conducted, as no eligible pairs exist. We contribute the TRIPOD-LLM-for-education checklist - to our knowledge the first reporting checklist derived from TRIPOD-LLM and calibrated for general (non-medical) education research - as a citable artefact, and call on education journals to require accountability reporting at submission.
No takes yet. Share an insight, caveat, or question.
Mintii et al. (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: