What question did this study set out to answer?

The research aims to assess the clinical reliability of large language models in providing patient guidance for invasive breast cancer based on established NCCN guidelines.

May 29, 2026

Evaluating reliability of large language models for patient queries: Concordance with NCCN invasive breast cancer guideline using ChatGPT, DeepSeek, and Gemini.

Key Points

The research aims to assess the clinical reliability of large language models in providing patient guidance for invasive breast cancer based on established NCCN guidelines.
Developed 62 patient-oriented questions aligned with NCCN guidelines across eight clinical domains.
Responses generated by ChatGPT, DeepSeek, and Gemini evaluated by three oncologists using a 5-point Likert scale.
Statistical analyses included inter-rater reliability assessment and Friedman’s chi-square test for differences between models.
Moderate inter-rater reliability for completeness (ICC = 0.39); low reliability for other metrics (ICC = 0.09–0.26).
DeepSeek excelled in completeness (mean 4.65) and factual accuracy (4.26), while ChatGPT was best in clarity (4.84).
Significant differences were found in completeness (χ² = 42.05, p < 0.001) and clarity (χ² = 19.12, p < 0.001) across models.

Abstract

e12550 Background: The increasing use of large language models (LLMs) by patients seeking healthcare information emphasizes the need to evaluate their clinical reliability. Invasive breast cancer, characterized by nuanced, multi-stage treatment protocols, represents a key domain where LLM-generated patient guidance must be assessed against established clinical standards. Methods: We developed 62 patient-oriented questions based on NCCN guidelines for patients on invasive breast cancer, spanning eight clinical domains: general information, testing, staging, types of treatment, postoperative outcomes, supportive care, treatment options, and recurrence. Responses were generated by ChatGPT (GPT-5.2), DeepSeek (V3.2), and Gemini (2.5 Flash). Three board-certified oncologists evaluated each response across five metrics—factual accuracy, clarity, coherence, relevance, and completeness—using a 5-point Likert scale. Inter-rater reliability was assessed with a two-way random-effects intraclass correlation coefficient (ICC2). Differences between models were analyzed using Friedman’s chi-square test and Wilcoxon signed-rank post-hoc tests, with descriptive statistics computed per domain. Results: Inter-rater reliability was moderate for completeness (ICC = 0.39) and low for other metrics (ICC = 0.09–0.26). Across 186 responses (62 questions × 3 models), DeepSeek led in completeness (mean 4.65) and factual accuracy (4.26), Gemini in coherence (4.79) and relevance (4.56), and ChatGPT in clarity (4.84). Friedman tests showed significant differences in completeness (χ² = 42.05, p ChatGPT in completeness (p DeepSeek in coherence (p = 0.048), and ChatGPT > DeepSeek in clarity (p < 0.001). Clinical domain analyses revealed variability: DeepSeek excelled in completeness for supportive care (4.73/5, n = 15) and treatment options (4.50/5, n = 10), outperforming ChatGPT (3.53 & 3.60) and Gemini (4.60 & 4.10). Conversely, ChatGPT and Gemini were superior in clarity/coherence for informational domains—general information about invasive breast cancer (both 5.00 clarity, 5.00/4.86 coherence, n = 7 vs DeepSeek 3.71) and testing for breast cancer (both ~5.00/4.80–5.00, n = 5 vs DeepSeek 4.60). This pattern shows the need for domain-tuned LLMs in clinical patient guidance. Conclusions: The quality of LLM-generated patient information is highly domain-dependent, with each model exhibiting distinct strengths. Low inter-rater reliability emphasizes the challenge of consistent evaluation. While LLMs can serve as supplementary educational tools, their clinical integration requires careful, domain-specific validation and clear guidance regarding their variable performance across different topics in breast cancer care.

KI fragen

Bookmark