Answers are grounded in this paper and cite their sources.
Objective Large language models (LLMs) are increasingly being used as revision tools. Radiation oncology encompasses a complex skill set, including clinical staging, trials, anatomy, radiobiology, physics, mathematical calculations, treatment planning, and quality assurance (QA). The accuracy, reasonability, and error types of two ChatGPT models were assessed across clinical and medical physics radiation oncology examination question banks. Methodology Two independent scoring studies were undertaken (n = 200). Cohort 1 assessed the accuracy of ChatGPT 5.5 across 100 single-best-answer clinical questions from the American College of Radiology (ACR) Radiation Oncology In-Training Examination (TXIT). Questions were selected at random using a computer-generated random sampling script (25 questions per year from the years 2021, 2024, 2025, and 2026). Cohort 2 assessed ChatGPT 5.6 (Luna build, v5.6) across 100 RAPHEX-style radiation oncology physics multiple-choice questions covering five domains of physics. Each question was asked once using a standardized zero-shot prompt requesting a single best answer with reasoning. Accuracy was scored against published answer keys. Response length, error types, and exploratory scoring of reasoning quality (0-2) were noted. Chi-square (χ²), Fisher's exact, Welch's t-test, and Mann-Whitney U tests were used to compare results (P = 0.05). Results ChatGPT 5.5 had an overall accuracy of 82.0% (82/100) (95% confidence interval (CI): 72.8%-89.0%). Accuracy was 100% for radiobiology, pediatrics, and renal/bladder oncology. Accuracy was lowest for gynecological oncology (female) at 60.0% and benign/palliative radiation oncology at 66.7%. Accuracy varied between years of examination (2021-2026) from 76.0% to 88.0% (χ² = 2.44, df = 3, P = 0.49). Incorrect responses had a greater mean length compared to correct responses (227.657.5 vs. 202.144.3) (Welch's t = 1.77, P = 0.09) (Mann-Whitney U = 551.5, P = 0.10). Error types were mainly guideline/trial/numerical-specific recall errors (5/18, 27.8%). ChatGPT 5.6 had an overall accuracy of 87.0% (87/100) (95% CI: 78.7%-93.2%). Accuracy was highest for brachytherapy (15/15, 100%) and imaging/planning (24/25, 96.0%). Accuracy was lowest for beam physics (21/27, 77.8%). Mean length of reasoning was 186.848.2 words (correct = 188.4, incorrect = 176.2) (Student's t = 0.81, P = 0.42). Error types were most commonly conceptual errors (5/13, 38%) and self-contradictory responses (4/13, 31%). Conclusions Both ChatGPT 5.5 and ChatGPT 5.6 had high agreement with the answer keys. However, error types differed. Clinical errors were mainly due to detailed guideline/trial recall and anatomical staging errors. Physics errors were often due to conceptual errors and mathematical contradictions. LLMs can be useful as revision tools; however, responses must be validated against authoritative literature.
No takes yet. Share an insight, caveat, or question.
Fahmy et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: