Abstract Study Objectives Large Language Models (LLMs) may efficiently develop Regular Expression (regex)-based Natural Language Processing (NLP) for electronic health record (EHR) phenotyping. We present PromptNLP to assess feasibility by evaluating performance and generalizability to clinical notes across multiple institutions. Methods Sleep-related clinical notes from Kaiser Permanente Southern California (KPSC), Beth Israel Deaconess Medical Center (BIDMC), and Emory University were used to extract apnea-hypopnea index (AHI) and Epworth Sleepiness Scale (ESS). PromptNLP involves two steps: (1) build an AHI/ESS phrase library (“Targeted” via manual, “General” via LLMs); (2) input the library into an LLM with prompts to generate regex. For model development (Phase 1), we compared PromptNLP regexes with manual regex. For external validation (Phase 2), we adapted Phase 1 Targeted regex to BIDMC and Emory notes. For application (Phase 3), we applied adapted regex to 207 404 KPSC notes with validation against 8602 structured AHI. Results Phase 1: All Targeted PromptNLP regexes outperformed manual regex for recall (AHI: 95–96% vs 80.8%; ESS: 98–100% vs 88%; p 0.05). General regexes surpassed manual, but with lower recall. Phase 2: Direct BIDMC application performed poorly (AHI/ESS: 30%/12%), but adapting the phrase library improved recall to 96%/84%. For Emory, recall improved from 85%/86% to 94%/89% after adaptation. Across Phase 1 and 2, PromptNLP regexes achieved 100% precision. Phase 3: 8216 (95.5%) AHI were correctly extracted with 99.8% precision. Conclusions PromptNLP demonstrated feasibility as an efficient, accurate, and generalizable method, outperforming manual approaches and enabling iterative cross-system improvement. Systematic comparisons to direct LLM approaches are needed.
Hwang et al. (Thu,) studied this question.