Automated Review of Patient Records: Privacy-Preserving LLMs for identifying incident NAION at Scale

Purpose: Retrospective identification of acute non-arteritic anterior ischemic optic neuropathy (NAION) cases is critical for research on risk factors.However, reliance on ICD-10 coding for case identification has limited accuracy, and manual review of longitudinal electronic health records (L-EHR) is time-intensive.The purpose of this study is to evaluate automated methods for retrospective identification of acute NAION cases using large language models (LLMs) that preserve patient privacy. Design: Retrospective cross-sectional studySubjects: 165 patients with ≥1 ICD-10 code for ION (H47.01*) in the electronic health record at an academic medical center Methods: Five locally deployed LLM models (Mistral Small 3.1, Magistral Small, Gemma3, MedGemma, GPT-OSS 20B) were used to implement four approaches for acute NAION diagnostic classification using unstructured ophthalmology records (basic prompting, retrievalaugmented generation (RAG), two-step agentic workflow, and three-step agentic workflow).10% of subjects were used for prompt refinement.LLM/approach diagnostic classifications were compared against expert neuro-ophthalmologist diagnosis based on chart review.Main outcome measures: positive predictive value (PPV) of LLM approaches for acute NAION case identification with expert chart review diagnosis serving as gold standard.Secondary outcomes included negative predictive value, sensitivity, specificity, accuracy, F1 score and distribution of LLM/approach classifications.