Can Language Models Diagnose Sepsis? A Head-to-Head Evaluation Against 353 Physicians

Importance: Sepsis is a leading cause of in-hospital mortality, where timely and accurate diagnosis is critical yet frequently delayed. Whether large language models (LLMs) can diagnose sepsis reliably -- and whether they do so fairly, consistently, and with well-calibrated confidence -- remains unknown. Objective: To evaluate the diagnostic capabilities of LLMs on standardised sepsis vignettes before and after biomarker disclosure, benchmarked against physicians, and to characterise their robustness, stability, bias and confidence calibration. Design: Retrospective in-silico study based on six anonymised real cases spanning the diagnostic spectrum from no infection to septic shock. For each case, the reference-standard diagnosis was assigned, following the Sepsis-3 framework. Each case was evaluated under a two-stage (pre- and post-biomarker disclosure) protocol. To assess potential demographic bias, every case was additionally presented while systematically varying the patient's sex, age, and language. Setting and Participants: Five locally deployed LLMs (DeepSeek-4, Qwen3.8-27B, its quantised variant, Gemma-4-31B, and the domain-specific MedGemma) were evaluated in a controlled evaluation pipeline and compared with responses from 353 clinicians, using identical clinical vignettes. Main Outcomes and Measures: Diagnostic accuracy before/after biomarker disclosure, per-vignette error, sensitivity to demographic attributes (diagnosis-change rate) and its directional component, reproducibility across repetitions and confidence calibration. Results: Before biomarkers, model accuracy ranged 40-67\%, comparable to physicians (57\%). After disclosure, physician accuracy was 61\%, exceeded by Qwen3.8-27B (91\%), its quantised variant (86\%) and Gemma-4-31B (83\%); DeepSeek-4 reached 64\% and MedGemma 56\%, the only model below physicians. Biomarkers rescued confirmatory cases (pancreatitis-associated sepsis), whereas post-surgical sepsis remained difficult for models and physicians alike. Quantisation cost only 5 accuracy points. Patient sex and age showed no directional bias, and language was the only bias. Gemma-4-31B was the most reproducible model, the Qwen models were the most accurate and best calibrated, and MedGemma the least. Conclusions and Relevance: LLMs, including quantised variants, can provide accurate informed decision support for sepsis, surpassing physicians once laboratory data are available. However, robustness, calibration, and language fairness varied independently of accuracy and should guide model selection and clinical validation.