Enhancing medical education with AI: A comparative study of various large language models in radiology

Objective To compare how various Large Language Models (LLMs) perform when solving text- and image-based medical student-level radiology multiple-choice questions (MCQs). Methods We evaluated the accuracy of seven LLMs (i.e., ChatGPT-4o, ChatGPT-5, ChatGPT-5 Plus, DeepSeek-V3.1-Terminus, Microsoft Copilot, Claude 4, and Gemini 2.5 Flash) in solving 60 text-based and 60 image-based MCQs. We compared the seven LLMs’ accuracy across question types, domains, and topics. Results The mean accuracy across the seven evaluated LLMs was 79.9% (671/840 correct responses). Claude 4 achieved the highest observed accuracy at 86.7%, whereas Gemini 2.5 Flash and Microsoft Copilot recorded the lowest accuracy at 75.0% each. Accuracy was substantially lower for image-based questions than for text-based questions (64.5% vs. 95.2%). In the generalized estimating equations analysis, question type was significantly associated with response accuracy (Wald χ 2 (1) = 64.53, p < 0.001), as were question domain (Wald χ 2 (1) = 6.86, p = 0.009) and the question type × domain interaction (Wald χ 2 (3) = 80.26, p < 0.001). The omnibus effect of model was not statistically significant after accounting for these factors (Wald χ 2 (6) = 6.59, p = 0.361). Although Claude 4 achieved the highest observed accuracy, no pairwise model comparison remained statistically significant after Bonferroni correction. Conclusions Large Language Models are less accurate at solving image-based than text-based radiology questions, supporting their role as adjunctive educational tools that require faculty oversight rather than independent replacements for radiology instruction.