553 Background: Large language models (LLMs) are increasingly used to support clinical reasoning in breast cancer (BC), yet reproducible benchmarking remains limited. In our prior work using complex breast cancer clinical scenarios (BCCS), OpenEvidence (OE) outperformed other LLMs; however, expert Likert-based grading may be influenced by subjective interpretation and human bias. Standardized, board-style multiple-choice questions—required even for expert clinicians—offer an objective framework to assess LLM accuracy and inform readiness for potential clinical decision support. We evaluated the accuracy of four LLMs on a standardized BC question set using binary grading and paired statistical testing. Methods: One hundred BC questions reflecting real-world clinical scenarios were submitted to four LLMs: ChatGPT-5.2 (GPT), Google Gemini 3 (Gemini), Claude Sonnet 4.5 (Claude), and OE. Questions were sourced from a hematology–oncology question bank and spanned major BC subtypes and treatment settings. Model responses were graded against bank-designated correct answers using binary scoring (1=correct, 0=incorrect). Overall accuracy with 95% confidence intervals (CI) was calculated. Pairwise comparisons were performed using McNemar’s test. Results: One hundred clinical scenarios were evaluated spanning all major BC subtypes and treatment settings, including early-stage and metastatic disease, and neoadjuvant and adjuvant therapy. Overall accuracy was highest for Gemini at 94.0% (95% CI, 87.4–97.8), followed by GPT at 90.0% (95% CI, 82.4–95.1), OE at 90.0% (95% CI, 82.4–95.1), and Claude at 89.0% (95% CI, 81.2–94.4). McNemar’s test showed no statistically significant differences between models. Compared with Gemini, GPT showed 4% lower accuracy (p=0.289), Claude 5% lower (p=0.267), and OE 4% lower (p=0.289). GPT and OE demonstrated equivalent accuracy (both 90.0%) with no significant difference (p=1.000). GPT vs Claude differed by 1% (p=1.000) and Claude vs OE differed by 1% (p=1.000). All models generated clinical reasoning, while OE was the only model providing citations. Conclusions: In standardized BC clinical scenarios, four widely used LLMs demonstrated high accuracy with no significant between-model differences. However, no model achieved 100% accuracy, reinforcing the need for ongoing refinement and clinician oversight prior to clinical use. Future studies should expand question volume, evaluate domain-specific error patterns, and incorporate question banks from multiple countries and geographic regions to improve generalizability before real-world clinical deployment.
Haji et al. (Wed,) studied this question.