Answers are grounded in this paper and cite their sources.
BACKGROUND: The modified Rankin scale (mRS) is an important metric in stroke research, often used as a primary outcome in clinical trials and observational studies. The mRS can be assessed retrospectively from electronic health records (EHR), but this process is labor-intensive and prone to inter-rater variability. Large language models (LLMs) have demonstrated potential in automating text classification. OBJECTIVE: We aim to create a fine-tuned LLM that can analyze EHR text and classify mRS scores for clinical and research applications. METHODS: We performed a retrospective cohort study of patients admitted to a specialist stroke neurology service at a large academic hospital system between August 2020 and June 2023. Each patient's medical record was reviewed at two time points: (1) hospital discharge and (2) approximately 90 days post-discharge. Two independent researchers assigned an mRS score at each time point. Two separate models were trained on EHR passages with corresponding mRS scores as labeled outcomes: (1) a multiclass model to classify all seven mRS scores and (2) a binary model to classify functional independence (mRS 0-2) versus non-independence (mRS 3-6). Four-fold cross-validation was conducted, using accuracy and Cohen's kappa as model performance metrics. RESULTS: A total of 2,290 EHR passages with corresponding mRS scores were included in model training. The multiclass model-considering all seven scores of the mRS-attained an accuracy of 77% and a weighted Cohen's Kappa of 0.92. Class-specific accuracy was highest for mRS 4 (90%) and lowest for mRS 2 (28%). The binary model-considering only functional independence vs non-independence -attained an accuracy of 92% and Cohen's Kappa of 0.84. Conclusions. CONCLUSIONS: Our findings demonstrate that LLMs can be successfully trained to determine mRS scores through EHR text analysis, however, improving discrimination between intermediate scores is required.
No takes yet. Share an insight, caveat, or question.
Silva et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: