557 Background: Second opinions and multidisciplinary reviews (MDR) are known to alter the management of cancer patients in a substantial percentage of cases, yet these resources are not universally available. We previously presented more than 400 anonymized complex cancer scenarios (cases) to MDR by experts in five major tumor types by video conference (2020-2021, www.precisca.com) and tracked treatment recommendations. Here, we compare MDR panel recommendations to those generated by 3 different foundational AI models. Methods: 50 complex cancer scenarios in breast and lung cancer that were previously adjudicated by MDR were analyzed by three different foundational AI models (OpenAI’s ChatGPT 4.5, Anthropic’s Claude Opus 4 and Google’s Gemini Ultra) using PrecisCa’s proprietary method for prompts. The AI recommendations from each system were scored on a scale of 1-5 (5 being the highest) for completeness, reasoning, clarity, menu of options, recency, and relevance as compared to the MDR panel recommendations. The maximum possible competence score was 30 points per scenario and an aggregate score of 750 for each tumor type. Final AI recommendations were also compared with current National Comprehensive Cancer Network (NCCN) guidelines for serious omissions. Comparison in reverse (additional AI options that the experts missed) was not performed due to interval changes in treatment recommendations over the past 4 years. Results: Patient characteristics and aggregate concordance and competence scores are listed in the table. While there was some variation in the concordance and competence rates of the three AI models, there was excellent concordance with the expert opinion for all of them. Discordant cases were reviewed and primarily involved minor differences that would not have altered the patients’ management significantly. Conclusions: This study demonstrates a high degree of concordance between three leading AI models and expert panel decisions for common yet complex breast and lung cancer clinical scenarios. These findings indicate that it is not premature to incorporate AI as a decision support tool in daily practice with human experts’ review. Updated data with additional cases and tumor types will be presented. Patient characteristics and aggregate competence score (n=50). Characteristic Breast Cancer (n=25) Lung Cancer (n=25) Median Age (Range) 55 (27-83) 70 (53-87) Stage I 5 (20%) 3 (12%) II 6 (24%) 1 (4%) III 2 (8%) 5 (20%) IV 12 (48%) 16 (64%) Histology Ductal Carcinoma 24 (96%) N/A Lobular Carcinoma 1 (4%) N/A Non-Small Cell N/A 22 (88%) Small Cell N/A 3 (12%) Aggregate / Median Competence Score (Range) OpenAI 4.5 649 / 25 (23-27) 626 / 25 (21-29) Claude Opus 4 708 / 27 (24-30) 694 / 27 (23-30) Gemini Ultra 720 / 28 (26-30) 727 / 28 (26-30)
Jahanzeb et al. (Wed,) studied this question.