Large Language Models for Accurate Medical Chart Abstraction: Enabling Scalable and Secure AI Deployment in Stroke

BACKGROUND AND PURPOSE: Medical chart abstraction plays a critical role in clinical research and quality monitoring by transforming unstructured narratives in the procedure reports into structured variables for large-scale analysis. To develop and evaluate a prompting-based large language model (LLM) framework for automated extraction of structured clinical variables from neurointerventional procedure reports in patients with acute ischemic stroke (AIS) due to large-vessel occlusions (LVO). MATERIALS AND METHODS: This retrospective study included 2,416 free-text neurointerventional acute stroke intervention (thrombectomy) reports with key radiology findings from three hospitals. Eight clinically relevant variables were annotated by hospital staff (without formal clinical training) and used as the non-expert reference standard. Twenty-two instruction-tuned open-source LLMs (LLaMA, Qwen, Gemma, etc) were evaluated across architectures, sizes, and biomedical adaptations using two prompting strategies: Quick Response and Chain-of-Thought (CoT). Model performance was benchmarked against non-expert staff annotations and medical expert ratings. Extraction accuracy, latency, and agreement with expert adjudication were assessed. RESULTS: LLaMA3.3-70B achieved the highest overall accuracy (94.8%). CoT prompting improved performance on inferential variables (e.g., site of occlusion), while Quick Response was optimal for directly stated procedural fields (e.g., stent placement). Expert adjudication confirmed that LLaMA3.3-70B outperformed non-expert annotations in 7 of 8 variables and matched junior medical students. Annotation accuracy increased with clinical experience, and AI predictions were more closely aligned with expert interpretations than with those of non-expert staff, especially for structured variables like IV tPA, TICI Post, and NIHSS. CONCLUSIONS: Prompted LLMs can accurately and scalably extract critical clinical information from neurovascular radiology reports without custom preprocessing, supporting integration into retrospective research pipelines and automated stroke registry curation.