854 Background: Multidisciplinary reviews (MDR) are known to alter the management of many cancer patients, yet these resources are not universally available. We previously presented more than 400 anonymized complex cancer cases spanning various tumor types including gastrointestinal cancers to a panel of experts in medical oncology, surgical oncology, radiation oncology, and radiology, for MDR. MDR treatment recommendations were captured. Here, we compare MDR panel recommendations to those generated by 3 different foundational AI models. Methods: We selected 48 complex GI cancer scenarios adjudicated between 2020 and 2021 by MDR panels from a larger database of multiple tumor types. These cases were analyzed by 3 different foundational AI models (OpenAI’s ChatGPT 4.5, Anthropic’s Claude Opus 4 and Google’s Gemini Ultra) using PrecisCa’s proprietary method for prompts. The AI recommendations from each system were scored on a scale of 1-5 (5 being the highest) for completeness, reasoning, clarity, menu of options, recency, and relevance as compared to those from the MDR panel. The maximum achievable score was 30 points per scenario and a total score of 1440. Final AI recommendations were also compared with current National Comprehensive Cancer Network (NCCN) guidelines. Comparison in reverse (additional AI options that the experts missed) was not performed due to interval changes in treatment recommendations over the past 4 years. Results: Patient characteristics and aggregate concordance and competence scores are listed in the table. When evaluating aggregate performance of all AI models across diseases, they performed best in esophageal cases and worst in colorectal ones. Within the 6 categories listed above, AI systems excelled in recency and less so in completeness. While there was some variation in the concordance and competence rates of the 3 AI models, there was good concordance with the expert opinion for all of them. Discordant cases were reviewed and primarily involved minor differences that would not have altered the patients’ management significantly. Conclusions: This study demonstrates a high degree of concordance between three leading AI models and expert panel decisions for common yet complex gastrointestinal cancer clinical scenarios. These findings indicate that it is not premature to incorporate AI as a decision support tool in daily practice with human experts’ review. Patient characteristics and aggregate competence score (n=48). Median Age (Range) 60 (32-88) Histology Anal carcinoma 3 (6.25%) Colorectal 21 (43.8%) Esophageal 6 (12.5%) Gastric 3 (6.25%) Hepatobiliary (cholangiocarcinoma, gallbladder, hepatocellular) 5 (10.4%) Pancreatic 10 (20.8%) Aggregate/Median Competence Score (Range) ChatGPT 4.5 1231 / 20.5 (11-30) Claude Opus 4 1249 / 20.5 (11-30) Gemini Ultra 1264 / 21.5 (13-30)
Jahanzeb et al. (Sat,) studied this question.