Answers are grounded in this paper and cite their sources.
Systematic review evaluates AI-generated questions in medical education, indicating comparable psychometric performance to human-generated items.
INTRODUCTION: Recently developed artificial intelligence (AI), particularly large language models (LLMs), e.g. GPT, streamline the development of multiple-choice questions (MCQs); however, concerns remain about psychometric quality, fairness across learner groups, and potential bias. METHODS: Following PRISMA 2020, we searched PubMed, Google Scholar, Web of Science, and the Cochrane Library for studies comparing both approaches in health education. We extracted difficulty and discrimination outcomes and assessed risk of bias using R Studio, the original Cochrane Risk of Bias tool for randomised trials and ROBINS-I for non-randomised studies. Random-effects meta-analyses were performed, with subgroup and sensitivity analyses to explore heterogeneity. Subgroup analyses by AI model and educational domain did not fully explain the significant difference between-study heterogeneity. However, human-generated questions were more difficult in medical licensing examinations. Sensitivity analysis revealed that excluding one study altered the main results, indicating a significant benefit for AI-generated items on discrimination. The evidence base is limited with little attention to equity in the literature. DISCUSSION (IMRAD): To our knowledge, this is the first review comparing AI versus human-generated MCQs. Our findings highlight that AI-generated MCQs demonstrate psychometric performance comparable to human items overall but are presently best positioned to supplement expert item development in lower-stakes contexts as research evolves. We propose a precautionary approach with staged, auditable safety checkpoints, including provenance, continuous psychometric surveillance, and human gatekeeping to support equitable, culturally safe, high-quality outputs.
No takes yet. Share an insight, caveat, or question.
Alani et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: