Answers are grounded in this paper and cite their sources.
Background/Objectives: Large language models (LLMs) are increasingly evaluated for empiric antibiotic decision support, but it remains unclear whether contemporary systems behave similarly under the same clinical and stewardship constraints. We compared four LLM configurations on real internal medicine admissions using a common, auditable stewardship benchmark. Methods: This single-center, retrospective, paired, non-interventional shadow-mode study included 486 admissions from a previously published 493-admission internal medicine cohort. For each admission, the clinician-prescribed empiric regimen was compared with recommendations from OpenAI GPT-5.5, Anthropic Claude Opus 4.7, Google Gemini 2.5 Pro, and Mistral Large 2512; OpenAI GPT-5.2 was analyzed separately as a supplementary legacy bridge. All LLMs received identical clinical content, local epidemiology context, antibiotic ontology, and three-field output schema. The primary endpoint was any contextually unsupported broad-spectrum escalation, defined by the study’s prespecified guardrail rules; the key secondary endpoint was a weighted contextual escalation penalty. Results: Clinician regimens met the primary endpoint in 16.7% of admissions. Rates were 13.0% for GPT-5.5 (paired risk difference [RD] −3.7 percentage points; Holm-adjusted p = 0.216), 15.0% for Gemini 2.5 Pro (RD −1.6; Holm-adjusted p = 0.509), 29.8% for Claude Opus 4.7 (RD +13.2; Holm-adjusted p < 0.001), and 31.9% for Mistral Large 2512 (RD +15.2; Holm-adjusted p < 0.001). Mean weighted contextual escalation penalty was 0.288 for clinician prescribing, 0.173 for GPT-5.5, 0.165 for Gemini, 0.510 for Claude, and 0.348 for Mistral. Global heterogeneity across contemporary LLMs was significant for both endpoints (p < 0.001). Conclusions: Contemporary LLM configurations were not interchangeable under the same stewardship benchmark. The benchmark measures stewardship-process performance (contextually unsupported escalation), not microbiological adequacy or missed coverage; fewer violations should therefore not be read as better overall therapy. These findings support configuration-specific, version-locked local evaluation before prospective shadow-mode and human-in-the-loop assessment.
No takes yet. Share an insight, caveat, or question.
Antonie et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: