Large Language Models Exhibit High Performance as Computable Phenotypes for Pediatric Asthma Exacerbation Detection in a Retrospective, Observational Study of Emergency Medical Services Encounters

OBJECTIVES: Pediatric asthma exacerbations are a common emergent condition treated by prehospital emergency medical services (EMS). However, retrospective identification of those patients for research, educational, and quality improvement purposes can be difficult given the unique structure of EMS' electronic health records. Computable phenotypes are algorithms used to identify patients of interest. The objective of this study was to explore the performance of large language models (LLMs) as pediatric asthma computable phenotypes for EMS data. METHODS: This is a retrospective, observational study testing the performance of state-of-the-art, open-source, general-purpose LLMs (Gemma-2, Llama-3.1-8B, Llama-3.3-70B, and Mistral-0.3), and one LLM specifically designed for medical use (OpenBioLLM), as pediatric asthma exacerbation computable phenotypes for EMS data. The goal of the phenotype was a binary classification of yes or no for an EMS encounter for a pediatric asthma exacerbation. We examined EMS patient encounter data from the ESO Data Collaborative between January 1, 2018 and December 31, 2021 for patients ages 2 - 18 years. Two pediatric emergency medicine physicians independently reviewed and annotated 1,000 encounters to label patients as pediatric asthma exacerbations. We tested models on structured and/or unstructured data, and explored basic and chain-of-thought prompts. We measured model performance using specificity, sensitivity, positive predictive value, negative predictive value, and macro F1. RESULTS: After applying the inclusion-exclusion criteria, 24,283 patient encounters remained. The median age was 12 years, with a slight majority (51%) of female patients. The best performing LLM overall was Llama 3.3's 70 billion parameter model using unstructured and structured data with 10-shot chain-of-thought prompts, with a F1 score of 0.894. Unstructured data alone gave the best F1 score for all models except Llama-3.3-70B and Mistral-0.3. Chain-of-thought prompts were more likely to produce better results than basic prompts, with 4 out of the 5 models giving their best performance when prompted by a chain-of-thought prompt. CONCLUSIONS: In this work we demonstrate that open source LLMs can be tailored for use as accurate prehospital pediatric asthma computable phenotypes using EMS data. To our knowledge, this represents the first LLM-based computable phenotypes trained on EMS data.