Comparison of Resident Physician Responses and Four Large Language Model Platforms on Obstetrics and Gynecology Questions

Background: Large language models (LLMs), including ChatGPT, Gemini, Microsoft Copilot, and DeepSeek, are increasingly used to answer complex clinical questions. Their accessibility and rapid development suggest potential value in medical education, residency training, and point-of-care decision support. However, direct comparisons with obstetrics and gynecology trainees remain limited. To compare the accuracy of multiple free and subscription-based LLMs with that of obstetrics and gynecology resident physicians across specialty-specific domains. Methods: A validated 40-item single-best-answer multiple-choice set covering endocrinology, gynecology, obstetrics, and oncology was developed by subspecialists and demonstrated good internal consistency (Cronbach’s alpha = 0.82). 25 residents completed the assessment. The same questions were posed to five LLMs: ChatGPT (GPT-4o Mini; free), ChatGPT (GPT-4o; subscription-based), Microsoft Copilot (GPT-4 based; free), DeepSeek (free), and Gemini (free). Results: Residents achieved a mean score of 20.6 out of 40 (51.5% correct). Overall accuracy was highest for ChatGPT (GPT-4o; subscription-based) at 90.0%, followed by DeepSeek and Gemini, each at 87.5%. ChatGPT (GPT-4o Mini; free) scored 47.5%, similar to residents, while Microsoft Copilot (free) scored 37.5%. In paired item-level comparisons using a two-sided exact McNemar test based on resident-majority item correctness (≥13/25 residents correct), ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini significantly outperformed residents (all p < 0.001), whereas ChatGPT (GPT-4o Mini; free) and Microsoft Copilot did not differ significantly (p = 1.000 and p = 0.607, respectively). Domain-specific analyses showed that ChatGPT (GPT-4o; subscription-based), DeepSeek, and Gemini generally outperformed residents in gynecology, obstetrics, and oncology, with smaller and less consistent differences observed in endocrinology. Qualitative review of incorrect outputs noted occasional internally inconsistent or ambiguous responses. Conclusions: Across a validated, curriculum-mapped 40-item question set, advanced LLMs achieved higher accuracy than residents, and in item-level paired comparisons using the resident-majority indicator, several models performed significantly better. Performance variability across models, including gaps between subscription-based and free versions and lower accuracy observed for some free tools, indicates that LLMs should complement, rather than replace, human expertise.