Enhancing the precision of lipid management quality assessment in ASCVD using large language models

Background Many health systems rely on structured electronic health record (EHR) data to evaluate adherence to low-density lipoprotein cholesterol (LDL-C) management guidelines¹ in patients with atherosclerotic cardiovascular disease (ASCVD). However, quality assessments based solely on structured EHR data fail to capture potential explanations for guideline-discordant care, such as external laboratory results or patient drug intolerance. Purpose To assess (1) whether Generative Pretrained Transformer 4 omni (GPT-4o), a large language model (LLM), could accurately extract pertinent data from free-text (unstructured) clinical notes and (2) the extent to which incorporation of these data would affect quality measures of lipid management. Methods We used structured EHR data to analyze LDL-C levels and statin use of patients with ASCVD seen in cardiology clinics at a large academic medical center from 2022-2024. We then applied GPT-4o with zero-shot, chain-of-thought prompting to free-text cardiology clinic notes to (1) identify external LDL-C values documented within the notes of patients without a recent in-system lipid panel and (2) infer reasons for statin non-use among those with LDL-C ≥70 mg/dL. We validated GPT-4o's accuracy through manual review of 500 randomly selected charts. Results Among 9,098 patients with ASCVD (median age 71.0 years, 61.6% male, 70.5% White), structured EHR data showed 7,122 (78.3%) had recent lipid panels, of whom 3,456 (48.5%) had LDL-C <70 mg/dL (Figure 1). Compared to manual chart review, GPT-4o demonstrated high precision in capturing external LDL-C values (0.97) and reasons for statin non-use, including intolerance (0.91), physician decision (0.96), patient refusal (0.90), or not documented (0.95). Among 1,976 patients without recent in-system lipid panels, GPT-4o identified 359 (18.2%) with external lipids documented in clinical notes, of whom 171 (47.6%) had LDL-C <70 mg/dL. Incorporating these external results led to a higher lipid testing rate (82.2% vs. 78.3%), though the proportion achieving LDL-C <70 mg/dL remained unchanged (48.5%; Figure 2). Among 3,854 patients with LDL-C ≥70 mg/dL, 2,895 (75.1%) were on a statin and 1,755 (45.5%) were on a high-intensity statin (HIS). GPT-4o identified 582 patients with documented statin intolerance. Removing these patients from the analysis improved overall rates of statin (81.9% vs. 75.1%) and HIS use (53.6% vs. 45.5%). Conclusion GPT-4o can retrieve and interpret clinically relevant data from free-text notes with high precision. While integrating this unstructured data yielded only modest changes in quality measures of lipid management, it revealed the clinical contexts behind apparent care gaps. LLM-augmented quality assessment may guide targeted quality improvement initiatives for patients with ASCVD by identifying specific barriers to optimal lipid management, including statin intolerance, patient reluctance, and clinical inertia.