What is the clinical evidence from this study?

Study design: Cross-Sectional. Population: Breast cancer (n=16). Intervention: OpenEvidence (OE) vs. ChatGPT-4o (GPT) and DeepSeek-R1 (DS). Primary outcome: Mean score on a 5-point Likert scale for clinical accuracy, clarity, relevance, and usability on treatment decision-making cases (Q1-9) (Cohen's d 1.12 vs GPT, 2.84 vs DS, p=<0.0001).

What question did this study set out to answer?

This evaluation aims to compare the performance of three large language models in aiding treatment decisions for breast cancer.

February 22, 2026

Abstract PS3-04-06: Benchmarking Large Language Models for Clinical Decision Support in Breast Cancer Care: A Multi-Institutional Expert Evaluation

Q: What are the key findings of this study?

OpenEvidence outperformed ChatGPT-4o and DeepSeek-R1 in breast cancer clinical decision-making, achieving the highest mean expert evaluation score (3.91 vs 3.19 and 2.93; P<0.0001).

Q: What question did this study set out to answer?

This evaluation aims to compare the performance of three large language models in aiding treatment decisions for breast cancer.

Key Result

OpenEvidence outperformed ChatGPT-4o and DeepSeek-R1 in breast cancer clinical decision-making, achieving the highest mean expert evaluation score (3.91 vs 3.19 and 2.93; P<0.0001).

Key Points

This evaluation aims to compare the performance of three large language models in aiding treatment decisions for breast cancer.
Proposed ten breast cancer clinical scenarios to three LLMs: ChatGPT-4o, DeepSeek-R1, and OpenEvidence.
Engaged 16 breast medical oncologists to grade the LLM responses on a 5-point Likert scale.
Applied repeated measures ANOVA to analyze model performance differences.
OpenEvidence achieved the highest mean score for treatment recommendations across various breast cancer subtypes.
ChatGPT-4o showed moderate performance, while DeepSeek-R1 lagged behind in clinical relevance and accuracy.
OpenEvidence was identified as a promising clinical decision support tool despite its verbosity.

Study Design

Type

Cross-Sectional (n=16)

Multicenter

Yes

Structured PICO

Does OpenEvidence outperform ChatGPT-4o and DeepSeek-R1 in providing clinical decision support for breast cancer cases?

Population

10 breast cancer cases mimicking real-world scenarios evaluated by 16 breast medical oncologists from 9 academic centers

Intervention

OpenEvidence (OE) large language model

Comparator

ChatGPT-4o (GPT) and DeepSeek-R1 (DS) large language models

Outcome

Performance graded using a 5-point Likert scale for clinical accuracy, clarity, relevance, and usability

OpenEvidence outperformed ChatGPT-4o and DeepSeek-R1 in generating guideline-concordant treatment choices for breast cancer clinical scenarios, highlighting its potential as a clinical decision support tool.

Main Result

Effect estimate: Cohen's d 1.12 vs GPT, 2.84 vs DS

Absolute Event Rate: 3.91% vs 3.19%

p-value: p=<0.0001

Limitations

OpenEvidence lacked multimodal image interpretation capabilities.
OpenEvidence verbosity may require streamlining.
DeepSeek-R1 relied on optical character recognition, limiting its utility in image-based scenarios.

Abstract

Abstract Background: Artificial intelligence (AI) and large language models (LLMs) are increasingly explored as tools to support clinical decision-making in oncology. However, evidence validating their performance in complex breast cancer clinical scenarios (BCCS) remains limited. Given breast cancer’s diverse subtypes, evolving standards of care, and the need for nuanced, personalized treatment, we compared three LLMs for treatment decision-making to assess their capabilities and determine their readiness for integration into real-world breast oncology clinics. Methods: Ten breast cancer cases mimicking real-world scenarios were posed to three LLMs: ChatGPT-4o (GPT), DeepSeek-R1 (DS), and OpenEvidence (OE). 16 breast medical oncologists (BMOs; assistant to full professors) from 9 academic centers graded responses using a 5-point Likert scale (1 = poor, 5 = excellent) for clinical accuracy, clarity, relevance, and usability. Questions(Q) 1-9 assessed treatment decision-making cases; Q10 tested multimodal image interpretation (MMI) skills. Repeated measures ANOVA evaluated model differences, followed by Tukey’s post hoc comparisons. Results: On BCCS Q1-9 spanning all major breast cancer subtypes and treatment settings, including early-stage, metastatic, neoadjuvant, and adjuvant, OE achieved the highest mean score (3.91 ± 0.48; 2.57-4.43), significantly outperforming both GPT (3.19 ± 0.67; 2.36-3.93) and DS (2.93 ± 0.54; 1.50-3.86) in overall performance (p 0.0001), with large effect sizes (Cohen’s d = 1.12 vs. GPT; d = 2.84 vs. DS). Repeated measures ANOVA identified significant differences among models in 5/9 (56%) Q (p 0.05), as shown in the Table. Pairwise comparisons showed OE outperformed GPT in 4/5 (80%) and DS in 5/5 (100%) significant BCCS. GPT modestly outperformed DS in 2/5 (40%) (Cohen’s d = 0.41). GPT showed the highest inter-reviewer variability (SD = 0.67 vs. 0.48-0.54), indicating less agreement among BMOs on its responses. OE provided well-supported treatment recommendations and recurrence risk assessments with citations, though it lacked MMI. DS relied on optical character recognition in Q10 (inflammatory breast cancer image; 2.93 ± 1.33), limiting its utility in image-based BCCS. GPT, the only vision-enabled model, scored highest (4.29 ± 1.14), highlighting its strong potential for MMI integration in breast cancer diagnostic workflows. Conclusions: This is the first comparison of GPT, DS, and OE in BCCS. OE generated the most guideline-concordant treatment choices across BCCS, showing strong potential as a clinical decision support tool, though its verbosity may require streamlining. GPT showed moderate performance, while DS lagged in clinical relevance and accuracy. These findings highlight the promise of LLMs in breast oncology and the need for further refinement to ensure reliability and real-world applicability. Citation Format: Z. Shah, S. S. Afridi, M. Ombada, A. M. Roy, A. LeVee, S. Premji, V. Gupta, N. M. Lopetegui, D. M. Quiroga, R. L. Sacks, S. Shaikh, Y. Abdou, H. Yu, A. Madabhushi, R. Parikh, L. N. Chaudhary, E. Levine, M. Lambertini, K. Kalinsky, S. Kabraji, S. Gandhi.. Benchmarking Large Language Models for Clinical Decision Support in Breast Cancer Care: A Multi-Institutional Expert Evaluation abstract. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-04-06.

Bookmark

Cite This Study

Shah et al. (Tue,) conducted a cross-sectional in Breast cancer (n=16). OpenEvidence (OE) vs. ChatGPT-4o (GPT) and DeepSeek-R1 (DS) was evaluated on Mean score on a 5-point Likert scale for clinical accuracy, clarity, relevance, and usability on treatment decision-making cases (Q1-9) (Cohen's d 1.12 vs GPT, 2.84 vs DS, p=<0.0001). OpenEvidence outperformed ChatGPT-4o and DeepSeek-R1 in breast cancer clinical decision-making, achieving the highest mean expert evaluation score (3.91 vs 3.19 and 2.93; P<0.0001).

synapsesocial.com/papers/699a9dcd482488d673cd3f13 https://doi.org/https://doi.org/10.1158/1557-3265.sabcs25-ps3-04-06

Bookmark