Answers are grounded in this paper and cite their sources.
Patients and clinicians increasingly use large language models (LLMs) for return-to-sport decisions, but their reliability and the effect of structured clinical context remain uncertain. In this pre-registered, blinded, multi-model study, GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and DeepSeek-V3.2-Exp answered 12 fictional anterior cruciate ligament reconstruction or lateral ankle sprain vignettes under instruction-only (Prompt+) and guideline-derived context (Context+) conditions across two runs, generating 192 outputs. Three blinded domain experts scored six binary criteria. The primary outcome was a correct clinical claim supported by a valid DOI or PMID. Context + did not improve the primary outcome compared with Prompt + (OR 0.41, 95% CI 0.16–1.05; p = 0.065). GPT-5 was the only model whose point estimate favoured Context+, whereas the other three models favoured Prompt+. Claude Sonnet 4.5 showed a higher majority-vote Hallucination rate under Context + (22.7% vs 62.5%; Holm-adjusted p = 0.014); however, inter-rater agreement for Hallucination was poor (Fleiss’ κ < 0 in both domains), so this finding should be interpreted cautiously. Citation validity declined under Context + in three of the four models. Structured guideline-derived context did not reliably improve return-to-sport decision support in the models and conditions tested, and its effects varied between models. LLMs intended for clinical decision support should therefore undergo task- and model-specific evaluation before deployment.
No takes yet. Share an insight, caveat, or question.
Gürses et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: