What question did this study set out to answer?

To evaluate and compare the accuracy, reliability, and consistency of five publicly available AI models across diverse question types and subjects on the USMLE Step 1 examination.

February 9, 2026JMIR AIOpen Access

Performance of 5 AI Models on United States Medical Licensing Examination Step 1 Questions: Comparative Observational Study

View Full Paper

Ask AI

Bookmark

Authors

DNDania El NatourBeirut Arab University MAMohamad Abou AlfaBeirut Arab University ACAhmad ChaabanUnited Arab Emirates University

Discussion

Member takes

Implication

Observational study reveals Grok outperforms four other AI models on USMLE Step 1 questions, highlighting rapid advancements in automated clinical reasoning.

Key Points

To evaluate and compare the accuracy, reliability, and consistency of five publicly available AI models across diverse question types and subjects on the USMLE Step 1 examination.
Cross-sectional observational study evaluating 119 USMLE-style questions (96 text-only and 23 image-based; 78 case-based and 41 information-based) across five AI models: Grok, ChatGPT-4, Copilot, Gemini, and DeepSeek.
Each model received standardized prompts and answered every question three separate times between February 10 and March 5, 2025, to assess consistency and self-correction.
Differences across categorical domains and question formats were evaluated using chi-square and Fisher exact tests with Bonferroni adjustments.
Overall question accuracy was highest for Grok (109/119, 91.6%), followed by Copilot (101/119, 84.9%), Gemini (100/119, 84.0%), ChatGPT-4 (95/119, 79.8%), and DeepSeek (86/119, 72.3%).
DeepSeek achieved 0% accuracy on image-based items due to lack of vision capabilities, but reached 89.6% (86/96) on text-only questions; Grok led in image-based items (21/23, 91.3%) and case-based items (70/78, 89.7%), significantly outperforming DeepSeek on case-based questions (P=.01).
Across all models, performance peaked in biostatistics and epidemiology (96.7%) and was lowest in musculoskeletal, skin, and connective tissue topics (62.9%), while Grok showed 100% response consistency across attempts.

Cite This Study

Natour et al. (2026) studied this question.

synapsesocial.com/papers/6a156444a2f71238514e608d https://doi.org/10.2196/76928

View Full Paper

Ask AI

Bookmark

Implication

Observational study reveals Grok outperforms four other AI models on USMLE Step 1 questions, highlighting rapid advancements in automated clinical reasoning.

Performance of 5 AI Models on United States Medical Licensing Examination Step 1 Questions: Comparative Observational Study

Discussion

Member takes

Implication

Key Points

Cite This Study

Performance of 5 AI Models on United States Medical Licensing Examination Step 1 Questions: Comparative Observational Study

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider