Beyond Billing Codes: An Agentic Large Language Model Pipeline for Extracting Glaucoma Procedural Data from Free-Text Clinical Notes

Purpose: To evaluate the accuracy of a large language model (LLM)-based agentic workflow to extract procedural data from free-text glaucoma clinical notes in the electronic health record (EHR). Design: Cross-sectional. Subjects: 1,250 subjects in the Bascom Palmer Ophthalmic Repository. Methods: Randomly sampled clinical notes from glaucoma encounters dated 2014 to 2024 were labeled by two glaucoma specialists (with an adjudicator as needed) for prior glaucoma-related procedures (name, date, details) and recommended future procedures (name, details). Notes were split into development (10%), validation (10%), and test (80%) sets. Five LLMs (Claude Opus 4.6, DeepSeek-V4 Pro, GPT-5.5, Grok 4.1, and Qwen3.6-35B-A3B) were accessed in Microsoft Foundry through HIPAA-compliant containers. The extraction pipeline comprised a three-step agentic workflow (extraction, validation, and conditional revision) followed by text standardization. Detection of procedure data was evaluated using accuracy, sensitivity, specificity, precision, and F1-score. Among positive cases, performance was evaluated using exact-match accuracy and Jaccard Index (JI). Prior procedure labels were also compared with EHR glaucoma-related Current Procedural Terminology (CPT) codes. Main Outcome Measures: Accuracy, exact-match accuracy, JI. Results: A total of 1,984 (992 notes * 2 eyes) labels were evaluated for each task in the test set. Prior and future procedures were identified in 1,057 (53.3%) and 152 (7.7%) eyes, respectively. Accuracy ranged from 0.965 (Grok) to 0.996 (Claude) for prior procedures and 0.969 (DeepSeek, Qwen)-0.978 (Claude) for future procedures. Among positive cases, exact-match accuracy ranged from 0.681 (Grok) to 0.890 (Claude) for prior procedures and 0.704 (Grok)-0.849 (Claude) for future procedures; JI ranged from 0.827 (Grok) to 0.956 (Claude) and 0.734 (Grok) to 0.883 (Claude), respectively. Among all 1,760 prior procedures present in adjudicated labels, only 441 (25.1%) were captured by CPT-coded EHR data. Conclusions: A secure agentic LLM pipeline accurately extracted glaucoma procedural data from clinical notes, recovering data largely absent from structured EHR fields. Claude performed best across both tasks. Limited agreement between labels and CPT codes likely reflects procedures performed at outside institutions and the limited granularity of CPT coding. This approach may support a scalable, comprehensive characterization of procedure burden among glaucoma patients for research and clinical applications.