What does this research mean for the field?

Modern large language models demonstrate proficiency in pediatric medical knowledge that generally exceeds medical student performance, though accuracy varies significantly across specific subspecialties. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

This analysis aims to evaluate the performance of leading AI language models on pediatric MCQs compared to medical students.

May 18, 2026Open Access

A Comparative Analysis of AI-Language Models’ MCQ Performance versus Medical Students Across Different Pediatric Topics

Key Points

This analysis aims to evaluate the performance of leading AI language models on pediatric MCQs compared to medical students.
Assessed 4 leading LLMs on 120 pediatric MCQs from six topics.
Each LLM attempted questions three times for consistency evaluation.
Medical student performance served as a benchmark against random responses.
LLMs achieved 80.4± 10.9% accuracy, outperforming students (72.1± 13.1%) by 8.3%.
Copilot (84.5± 8.3%) and Claude (84.2± 9.9%) had the highest accuracy.
Only GPT-4o showed a statistically significant accuracy difference (χ² = 4.277, p = 0.039), outperforming students by 8.8%.

Abstract

Background: Large Language Models (LLMs) are increasingly used in medical education, yet their performance in specialized fields like pediatrics remains understudied. Objective: To evaluate and compare the performance of four leading LLMs on a standard set of pediatric multiple-choice questions (MCQs) and benchmark their results against those of medical students. Methods: We assessed 4 leading LLMs: Copilot (Microsoft), Claude (Anthropic), ChatGPT (OpenAI), and Gemini (Google), on 120 MCQs from six pediatric topics: Pulmonology, Developmental Diseases, Infectious Diseases, General Pediatrics, Neonatology, and Nephrology. Each LLM attempted the questions three times to evaluate consistency. Medical student performance on the same questions served as a benchmark, with random responses providing baseline control. Results: The LLMs demonstrated an average accuracy of 80.4± 10.9%, outperforming medical students (72.1± 13.1%) by 8.3% and exceeding random responses (20.5± 6.2%) by nearly fourfold. Copilot (84.5± 8.3%) and Claude (84.2± 9.9%) achieved the highest accuracy, followed by GPT-4o (80.9± 9.1%) and Gemini (72.0± 18.9%). Among all LLM-to-student comparisons, only GPT-4o showed a statistically significant difference in accuracy (χ 2 = 4.277, p = 0.039), outperforming students by 8.8%. LLMs excelled in Pulmonology (96.3%), Developmental Diseases (86.3%), and Infectious Diseases (85%) while struggling with Nephrology (67.5%) and Neonatology (69.7%). Only 50% (60/120) of questions were correctly answered by all LLMs across all attempts, while 6.7% (8/120) were never answered correctly. Conclusion: Modern LLMs demonstrate proficiency in pediatric knowledge that generally exceeds medical student performance, though with varying consistency across topics. These findings suggest LLMs may serve as valuable supplementary tools in medical education while highlighting the need for further improvements in specialized medical domains like nephrology and neonatology. Keywords: artificial intelligence, medical education, pediatrics, large language models, multiple choice questions

Ask AI

Helpful

Bookmark

View Full Paper