What question did this study set out to answer?

This research aims to assess the effectiveness of generative AI models, Gemini and Copilot, in generating assessment questions for medical education.

March 21, 2026Open Access

Pushing the boundaries of generative AI: multiple-choice question generation and assessment performance within medical education

Key Points

This research aims to assess the effectiveness of generative AI models, Gemini and Copilot, in generating assessment questions for medical education.
Generated 335 MCQs from two virtual patient cases using standardized prompts.
Evaluated the generated MCQs based on alignment with performance criteria and educational standards.
Used expert assessments to determine the quality of the MCQs and the tools' performance.
Analyzed performance using reliability and agreement statistics.
Gemini's MCQs showed superior consistency with expert consensus in Miller classifications.
Copilot's MCQs excelled in Bloom's alignment with expert judgments.
Both AI tools aligned well with learning objectives and key identifications.
Experts found MCQs easier than AI assessments suggested, revealing discrepancies in perceived difficulty.

Abstract

Aims: The aim of this study is to systematically evaluate the performances of large language model-based generative Artificial Intelligence (Gen-AI) tools, Gemini and Copilot, in the generation and assessment of multiple-choice questions (MCQs) for use in medical education.Methods: A total of 335 MCQs were generated from two virtual patient cases using standardized prompts. Gen-AI tools selected the 56 best-quality items based on criteria encompassing the intended distributions regarding acceptable level of performance (ALP), Miller's competency pyramid (Miller) and Bloom's revised taxonomy (Bloom) levels, as well as alignment with learning objectives (LOs). Expert medical educators and current Gen-AI tools assessed these items based on the identification of misleading/confusing distractor(s) for borderline candidates -minimally competent examinees- (to calculate ALP values) and the identification of key(s), as well as Miller and Bloom levels, LO alignment, stem appropriateness, and technical item flaws. "AI-extended consensus" served as intersubjective consensus model (the gold standard). Generation performance was quantified by alignment with this consensus, and assessment performance by the degree to which Gen-AIs shifted or preserved Expert assessments. Analyses included ICC for reliability, Po/Cohen’s/Fleiss’ Kappa for categorical agreement, and inferential tests (Exact McNemar and Wilcoxon signed-rank) for detecting systematic bias and directional shifts.Results: Gen-AIs demonstrated markedly different performance patterns in assigning cognitive levels. For Miller, Gemini generated MCQs exhibited superior consistency with the intersubjective consensus (ICC(2,k)=0.82), whereas for Bloom, Copilot-generated MCQs demonstrated this superiority (ICC(2,k)=0.97). Both tools performed well in LO alignment and key identification, but their approaches to stem structure diverged substantially. Experts perceived the MCQs to be easier than the Gen-AIs claimed, and the current Gen-AI versions found them even easier than both the generating versions and the Experts did. In terms of assessment behaviour, Gen-AIs showed a systematic stringency tendency in Miller classifications, statistically significantly shifting Expert consensus from 'knows' to 'knows how' (p

KI fragen

Bookmark

View Full Paper