What question did this study set out to answer?

June 26, 2026Open Access

Performance variation and implementation barriers of large language models in clinical healthcare: a systematic review

Key Points

This review aims to assess the effectiveness and limitations of large language models in clinical healthcare, focusing on their performance and deployment challenges.
Systematic review of studies from January 2020 to January 2025
Eligibility for studies included transformer-based large language models with ≥10M parameters in clinical settings
Quality assessment using QUADAS-2, RE-AIM, and TRIPOD frameworks, with narrative synthesis per SWiM guidelines
Domain-adapted models achieved 88–98% accuracy on narrow tasks compared to 78–91% for general-purpose models
Real-world performance declined by 5–28% across clinical environments
Hallucination rates were 5–12% for domain-adapted models and 15–30% for general-purpose models in generative tasks

Abstract

Abstract Large language models (LLMs) are a rapidly evolving class of artificial intelligence with significant potential in clinical healthcare. Despite accelerating adoption, rigorous systematic evidence on clinical utility, patient safety, and implementation feasibility remains fragmented. To systematically review LLM applications across clinical domains, evaluate performance with appropriate contextual caveats, characterize implementation barriers, and identify ethical and regulatory considerations. Scientific databases were searched from January 2020 to January 2025. Studies evaluating transformer-based LLMs (≥10M parameters) in clinical settings were eligible. Data were independently double-extracted; quality was assessed using QUADAS-2, RE-AIM, and TRIPOD frameworks. Due to substantial heterogeneity across domains, narrative synthesis was conducted per SWiM guidelines; descriptive statistics are presented for the one sufficiently homogeneous domain (clinical documentation, domain-adapted models, n=12). Fifty-two studies were included. Domain-adapted models (ClinicalBERT, BioBERT, Llama-3-8B) outperformed general-purpose models (GPT-4, Med-PaLM 2) on structured, narrow tasks in benchmark settings (88–98% vs. 78–91% accuracy). These figures derive from curated datasets and should not be extrapolated to routine clinical environments. Across 34 studies reporting both benchmark and deployment data, real-world performance declined consistently (5–28% reduction). Hallucination rates were 5–12% for domain-adapted and 15–30% for general-purpose models in generative tasks. Key barriers included data privacy concerns (89%), absent regulatory frameworks (77%), and limited interpretability (83%). LLMs show promise in controlled settings, but evidence is dominated by retrospective evaluations on curated datasets and real-world performance is consistently lower. Responsible clinical integration requires addressing reliability, interpretability, privacy, regulatory readiness, and demographic equity.

Mark Helpful

Bookmark

Relay

View Full Paper