Development and Validation of a Large Language Model Case Identification Strategy for Eosinophilic Esophagitis

Background and Aims: Epidemiologic research in eosinophilic esophagitis (EoE) is limited by the accuracy and efficiency of case identification algorithms. We aimed to evaluate rule-based natural language processing (RB-NLP) and large language model–based natural language processing (LLM-NLP) pipelines for identifying EoE diagnoses and features from unstructured text. Methods: We identified gastrointestinal pathology reports with any mention of “eosinophil” paired with gastroenterology clinic notes. Three hundred randomly selected patients were divided into training (n = 200, 56 with EoE) and testing (n = 100, 36 with EoE) sets. Manual chart review was the reference standard. RB-NLP used spaCy with medspaCy’s clinical components; LLM-NLP prompts were developed through iterative human-in-the-loop refinement. In the validation set, we compared International Classification of Diseases (ICD) codes, RB-NLP, and LLM-NLP against the reference standard using sensitivity (recall), positive predictive value (precision), and F1 score. Results: In the validation set, ICD codes alone had a sensitivity 0.86 (95% confidence interval [CI]: 0.75–0.97), a positive predictive value of 0.97 (95% CI: 0.91–1.0), and an F1 value of 0.91 (95% CI: 0.84–1.0). Combining ICD and LLM-assigned diagnosis yielded a 3-point improvement in F1 score (95% CI: −0.01 to 0.07; P = .2) compared to ICD alone. In a larger cohort (n = 580), the LLM + ICD approach identified the most EoE cases (n = 203) and captured 15% of cases missed by ICD codes. Clinical characteristics varied depending on the case identification strategy used. Conclusion: Combining LLM-NLP with a single ICD code reduced false negatives and modestly improved the F1 score compared to either method alone. This may represent a scalable approach to enhance EoE case identification in real-world data.