Key result
GPT-4o achieves ~98% accuracy classifying stroke types from unstructured EHRs but struggles with ischemic subtypes.
Why the study?
Does GPT-4o accurately identify stroke types and subtypes from unstructured clinical notes compared to trained abstractors in patients with stroke hospitalizations?
Observational (n=4,123)
Yes
Does GPT-4o accurately identify stroke types and subtypes from unstructured clinical notes compared to trained abstractors in patients with stroke hospitalizations?
Answers are grounded in this paper and cite their sources.
GPT-4o demonstrated high accuracy in classifying stroke types from unstructured EHR data, though it faced challenges with ischemic subtypes.
BACKGROUND: While International Classification of Diseases, Tenth Revision codes suffice for identifying stroke events in surveillance, accurately classifying stroke types and subtypes using electronic health records remains challenging due to limitations in structured data. This often necessitates manual review of clinical documentation. This study evaluated whether a large language model, Generative Pre-Trained Transformer 4 Omni (GPT-4o), can accurately identify stroke types and subtypes from unstructured clinical notes. METHODS: We implemented a retrieval-augmented generation framework with GPT-4o to classify stroke types (ischemic versus hemorrhagic) and ischemic stroke subtypes using electronic health records data. The American Heart Association Get With The Guidelines–Stroke registry served as the gold standard. Model development used a 20% subset of Get With The Guidelines–Stroke–linked data from UT Southwestern Medical Center (UTSW), with the remaining 80% reserved for testing. External validation used data from the Parkland Health and Hospital System (PHHS). A total of 4123 stroke hospitalizations from January 2019 to August 2023 were included (UTSW: n=2047; PHHS: n=2076). Three prompting strategies—zero-shot chain-of-thought, expert-guided, and instruction-based—were evaluated. Predictions of GPT-4os were compared with classifications made by trained abstractors contributing to the Get With The Guidelines–Stroke registry. RESULTS: In the external validation set, 79.6% of patients had ischemic stroke and 20.4% hemorrhagic. GPT-4o achieved 98% accuracy (95% CI, 0.97–0.99) in classifying stroke type, where accuracy reflects the overall proportion of correctly classified patients. Sensitivity was 0.98 (95% CI, 0.97–0.99), and specificity was 0.97 (95% CI, 0.96–0.98). For ischemic stroke subtypes, sensitivity ranged from 0.40 (95% CI, 0.31–0.49) for cryptogenic to 0.95 (95% CI, 0.93–0.97) for small-vessel occlusion. Specificity ranged from 0.94 (95% CI, 0.92–0.96) for large-artery atherosclerosis to 0.98 (95% CI, 0.97–0.99) for cardioembolism. Zero-shot chain-of-thought prompting—requiring minimal human input—performed comparably to more labor-intensive strategies. Consistency analysis revealed > 99% agreement across repeated queries. CONCLUSIONS: GPT-4o demonstrated strong accuracy in classifying stroke types but faced challenges with ischemic subtypes.
No takes yet. Share an insight, caveat, or question.
Owens et al. (2025) conducted an observational in Stroke (n=4,123). Generative Pre-Trained Transformer 4 Omni (GPT-4o) vs. Trained abstractors (Get With The Guidelines-Stroke registry) was evaluated on Accuracy in classifying stroke type (ischemic versus hemorrhagic) (95% CI 0.97-0.99). GPT-4o achieved 98% accuracy in classifying stroke types from unstructured electronic health records, but faced challenges with ischemic subtypes.
Synapse has enriched 10 related papers. Consider them for comparative context: