Answers are grounded in this paper and cite their sources.
Purpose Assessment is crucial in health professional education (HPE) but is resource-intensive to create. Large language models (LLMs) offer a potential solution by generating assessment items; however, a comprehensive synthesis of their psychometric quality is lacking in HPE. Design/methodology/approach We systematically searched the electronic databases and included studies generating assessment items using any LLM and reporting psychometric outcomes. The primary outcomes were item difficulty index, discrimination index, distractor effectiveness and risk of nonfunctioning distractors (NFDs). Pooled estimates (Hedges' g or odds ratios [OR] with 95% confidence intervals [CIs]) were calculated using a random-effects model. Surface under the cumulative ranking curve (SUCRA) for ranking the test item generating methods was estimated. Grades of Recommendation, Assessment, Development and Evaluation approach was used for determining the strength of evidence. Findings Fifty-four studies were included. In direct comparisons with human-generated items (92.8% of studies in HPE), LLM-generated items had a significantly higher difficulty index (pooled g: 0.40, 95% CI: 0.23 to 0.57), indicating they were easier and had significantly higher odds of containing NFDs (OR: 1.86, 95% CI: 1.27 to 2.73). The discrimination index was not significantly different (g: –0.21, 95% CI: –0.49 to 0.07), but distractor effectiveness was poorer (g: –0.65, 95% CI: –1.31 to 0). When evaluated alone, 67% of LLM-items were accurate or required only minor revisions, but 37% of distractors were NFDs. SUCRA analyses consistently ranked human-generated items as superior across all critical psychometric parameters. Evidence certainty ranged between very low and moderate. Originality/value LLMs can generate assessment items efficiently, but their current output is psychometrically inferior to human-authored items, being significantly easier and containing more flawed distractors. While a substantial proportion of items are useable with minor edits, sophisticated prompt engineering and expert-led review are essential to ensure the creation of valid and reliable assessments for HPE.
No takes yet. Share an insight, caveat, or question.
Sridharan et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: