Empowering Non-Oncology Providers: Enhancing Controversial Surgical Oncology Decision-Making with Generative AI

Background Generative AI, including large language models (LLMs), increasingly supports clinical guideline retrieval and decision-making. However, surgical oncology decisions are complex and multidisciplinary, especially when specialized tumor boards are unavailable. This study evaluated ChatGPT's ability to generate recommendations for debated surgical oncology scenarios, identifying strengths and limitations in decision-making. Methods Twelve clinical vignettes representing debated surgical oncology scenarios were entered into ChatGPT (version 4.1). Responses were independently evaluated by four fellowship-trained surgical oncologists and one surgical oncology fellow using a 10-domain Likert-scale rubric. Descriptive statistics and a linear mixed-effects model evaluated performance across domains while accounting for rater and scenario variability. Results ChatGPT performed best in relevance (mean ± SD: 4.0 ± 0.9) and accuracy (3.8 ± 1.0), with moderate alignment with standard of care (3.0 ± 0.8). Performance was significantly lower for articulation of risks (2.1 ± 1.2), alternatives (2.2 ± 1.1), benefits (2.7 ± 1.2), and uncertainty (2.5 ± 1.1). Mixed-effects modeling confirmed significantly higher scores for relevance and accuracy and considerably lower scores for risks, alternatives, benefits, and uncertainty (all p ≤ 0.03). Performance was strongest in guideline-driven scenarios and weakest in clinically ambiguous cases. Conclusions ChatGPT reliably synthesizes guideline-aligned information but demonstrates limitations in the higher-order reasoning required for individualized surgical oncology decisions. These findings support its use for education or preliminary support while reinforcing the need for expert oversight before clinical integration.