Key result
GPT-o1 achieves ~98% accuracy classifying aortic stenosis severity using chain-of-thought prompting.
Why the study?
Aortic stenosis assessment requires integrating discordant echocardiographic and clinical information, and the reliability of large language models across different prompts and AS phenotypes remained uncertain.
Does large language model and prompting strategy selection affect accuracy in aortic stenosis severity classification, concordance with real-world valve intervention, and prognostic stratification in patients with aortic stenosis?
Cohort (n=1,455)
Yes
Does large language model and prompting strategy selection affect accuracy in aortic stenosis severity classification, concordance with real-world valve intervention, and prognostic stratification in patients with aortic stenosis?
LLM performance in aortic stenosis classification and management recommendation varies substantially across models, prompts, and rare discordant AS phenotypes, underscoring the need for prompt standardisation and clinician supervision before clinical deployment.
Answers are grounded in this paper and cite their sources.
LLM aortic stenosis classification requires clinician oversight before use; leaves open whether model recommendations independently affect outcomes.
Objectives Aortic stenosis (AS) assessment requires integration of discordant echocardiographic and clinical information. The reliability of large language models (LLMs) across prompts and AS phenotypes remains uncertain. This study aimed to compare five LLMs across four prompting strategies for AS severity classification, real-world intervention concordance and exploratory prognostic associations.Methods This retrospective study evaluated five LLMs across four prompting strategies in 1455 patients with AS. Task 1 assessed AS severity classification and Task 2 assessed real-world intervention concordance. Performance was summarised by accuracy, balanced accuracy and macro-F1. Exploratory associations of the best performing Task 2 configuration with all-cause mortality and heart failure progression were examined using Kaplan-Meier analysis and multivariable Cox regression.Results GPT-o1/chain-of-thought (CoT) achieved the highest Task 1 accuracy (97.53%), balanced accuracy (90.93%) and macro-F1 (83.19%). Grok-2/CoT achieved the highest Task 2 concordance (88.32%). Grok-2/CoT recommendations separated heart failure progression-free survival in unadjusted analysis (p=0.006) but not overall survival (p=0.105). After adjustment, recommendation group was not independently associated with mortality (HR 1.41, 95% CI 0.58 to 3.45, p=0.453) or heart failure progression (HR 1.10, 95% CI 0.44 to 2.74, p=0.832).Discussion LLM performance varied by model, task and prompt. Apparent prognostic separation observed in unadjusted analysis was not retained after adjustment for clinical characteristics, AS severity and actual intervention.Conclusion LLMs require prompt standardisation, class-specific evaluation and prospective validation before clinical use.
No takes yet. Share an insight, caveat, or question.
Huang et al. (2026) conducted a cohort in Aortic stenosis (n=1,455). Five large language models (GPT-4o, GPT-4, GPT-o1, DeepSeek-R1, Grok-2) evaluated across four prompting strategies (zero-shot, few-shot, chain-of-thought, meta-prompting) vs. Expert consensus reference standard (senior cardiologists and multidisciplinary heart team based on 2021 ESC/EACTS guidelines) and observed real-world aortic valve intervention was evaluated on AS severity classification accuracy (Task 1) and real-world intervention concordance (Task 2) (95% CI 96.59 to 98.21, p=<0.001). GPT-o1 with chain-of-thought achieved 97.53% accuracy in AS severity classification, while Grok-2 achieved 88.32% intervention concordance without independent prognostic value after adjustment.
Synapse has enriched 10 related papers. Consider them for comparative context: