Answers are grounded in this paper and cite their sources.
Background/Objectives: Large language models (LLMs) are increasingly used to obtain medical information, but the quality of their responses to complex, time-sensitive neurovascular questions requires disease-specific evaluation. We compared the expert-rated response quality of five publicly accessible LLM systems using standardized questions on aneurysmal subarachnoid hemorrhage (aSAH). Methods: Thirty standardized Turkish-language questions covering aneurysm treatment, vasospasm and related complications, and neurocritical care and follow-up (10 questions per domain) were submitted to ChatGPT, Gemini, Claude, DeepSeek, and Microsoft Copilot through public web interfaces in July 2026. Each question was submitted once per system in a separate chat session, generating 150 responses. Five senior neurosurgeons independently rated each response on a 1–10 scale for coverage, relevance, comprehensibility, clinical applicability, and overall quality. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Repeated-measures linear mixed models, Bonferroni-adjusted pairwise comparisons, and Friedman sensitivity analyses were performed. Results: The evaluation comprised 3750 expert score observations without missing data. Inter-rater reliability was good to excellent (ICC 0.835–0.911). AI system was significantly associated with all five evaluation dimensions (all p < 0.001). Adjusted overall scores were highest for ChatGPT (7.68), followed by Gemini (6.97), Claude (6.69), DeepSeek (5.92), and Copilot (4.75). Neither the clinical-topic main effect nor the model-by-topic interaction was statistically significant (all p > 0.05). Friedman analyses also identified between-model differences across all dimensions (all p < 0.001). Conclusions: Publicly accessible LLM systems differed significantly in expert-rated response quality when answering standardized Turkish-language aSAH questions. These differences were observed within the study’s specific question set and evaluation protocol and should not be interpreted as evidence of differences in real-world clinical outcomes. The absence of statistically significant model-by-topic interactions does not establish equivalent performance across clinical domains. Model-specific validation, assessment of clinically important errors, and continued expert oversight are needed before considering LLM-generated information for clinical decision support.
No takes yet. Share an insight, caveat, or question.
Taşkın et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: