Item-writing flaws and answer-position bias in large language model-generated medical multiple-choice questions: a benchmark study of 15 models with an open corpus and a rule-based detector

PURPOSE: Item-writing flaws in large language model (LLM)-generated medical multiple-choice questions have been assessed in small samples. This study developed an open benchmark of such items labeled for structural flaws, compared 15 models and 2 prompts, traced the source of answer-position bias, and evaluated LLM judges against physicians. METHODS: Fifteen LLMs generated 12,000 vignette items across 8 specialties using a plain prompt and a prompt with 10 item-writing guidelines, plus 15,800 items in 5 experimental arms. A 14-rule detector labeled the corpus and 667 retired National Board of Medical Examiners (NBME) items; it was validated against 3,737 external human labels and 2 independent readers. Two physicians rated 100 items; 4 LLM judges assessed those items and 216 human-labeled items. RESULTS: The detector found no format flaws, but 23.4% of plain-prompt items had a structural flaw, compared with 17.3% of NBME 5-option items. Excluding the longest-option rule reduced the gap from 6.1 to 1.2 percentage points. Guidelines halved the odds of a flaw (odds ratio, 0.50; 95% confidence interval, 0.45-0.56), reducing prevalence to 13.6%. Under the plain prompt, 7 models placed the key at position A in 61%-94% of items; removing the template's example letter reduced the pooled proportion from 54% to 23%. Physicians found a content defect in 12%-13% of items (κ=0.59). The best LLM judge achieved agreement of κ=0.53-0.56 with the physicians on differently defined composites, a preliminary result. CONCLUSION: LLM-generated items avoided format flaws but contained answer cues and a position-A bias arising mostly from the output template. Guidelines reduced the flaws they specified without demonstrably improving content, which needs physician review.