Blinded comparative study reveals equivalent psychometric quality between GPT-4 and human test items in medical students, indicating artificial intelligence can assist examination design.
PURPOSE: Creating clinically complex Multiple Choice Questions (MCQs) for medical assessment can be time-consuming . Large language models such as GPT-4, a type of generative artificial intelligence (AI), are a potential MCQ design tool. Evaluating the psychometric properties of AI-generated MCQs is essential to ensuring quality. METHODS: A 120-item mock examination was constructed, containing 40 human-generated MCQs at novice item-writer level, 40 at expert level, and 40 AI-generated MCQs. int. All examination items underwent panel review to ensure they tested higher order cognitive skills and met a minimum acceptable standard. The online mock examination was administered to Australian medical students, who were blinded to each item's author. CONCLUSIONS: The psychometric properties of AI-generated MCQs are comparable to human-generated MCQs at both novice and expert level. Item quality can be improved across all author groups. AI-generated items should undergo human review to enhance distractor efficiency.
No takes yet. Share an insight, caveat, or question.
Pham et al. (2025) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: