What does this research mean for the field?

Open-source large language models, particularly when utilizing chain-of-thought prompting and unstructured data, can accurately function as computable phenotypes to identify pediatric asthma exacerbations from emergency medical services (EMS) encounter records. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.ESTABLISHES_NEW_DIRECTION.

What question did this study set out to answer?

This study explores large language models as computable phenotypes for identifying pediatric asthma exacerbations in EMS data.

June 7, 2026

Large Language Models Exhibit High Performance as Computable Phenotypes for Pediatric Asthma Exacerbation Detection in a Retrospective, Observational Study of Emergency Medical Services Encounters

Key Points

This study explores large language models as computable phenotypes for identifying pediatric asthma exacerbations in EMS data.
Retrospective observational study utilizing EMS data from January 1, 2018 to December 31, 2021.
Analyzed 24,283 encounters using various large language models (Gemma-2, Llama-3.1-8B, Llama-3.3-70B, Mistral-0.3, OpenBioLLM).
Performance measured using specificity, sensitivity, positive predictive value, negative predictive value, and macro F1 scores.
Llama-3.3-70B achieved the best F1 score of 0.894 with chain-of-thought prompts.
Unstructured data yielded the highest F1 scores for most models except Llama-3.3-70B and Mistral-0.3.
Chain-of-thought prompts generally outperformed basic prompts in enhancing model accuracy.

Abstract

OBJECTIVES: Pediatric asthma exacerbations are a common emergent condition treated by prehospital emergency medical services (EMS). However, retrospective identification of those patients for research, educational, and quality improvement purposes can be difficult given the unique structure of EMS' electronic health records. Computable phenotypes are algorithms used to identify patients of interest. The objective of this study was to explore the performance of large language models (LLMs) as pediatric asthma computable phenotypes for EMS data. METHODS: This is a retrospective, observational study testing the performance of state-of-the-art, open-source, general-purpose LLMs (Gemma-2, Llama-3.1-8B, Llama-3.3-70B, and Mistral-0.3), and one LLM specifically designed for medical use (OpenBioLLM), as pediatric asthma exacerbation computable phenotypes for EMS data. The goal of the phenotype was a binary classification of yes or no for an EMS encounter for a pediatric asthma exacerbation. We examined EMS patient encounter data from the ESO Data Collaborative between January 1, 2018 and December 31, 2021 for patients ages 2 - 18 years. Two pediatric emergency medicine physicians independently reviewed and annotated 1,000 encounters to label patients as pediatric asthma exacerbations. We tested models on structured and/or unstructured data, and explored basic and chain-of-thought prompts. We measured model performance using specificity, sensitivity, positive predictive value, negative predictive value, and macro F1. RESULTS: After applying the inclusion-exclusion criteria, 24,283 patient encounters remained. The median age was 12 years, with a slight majority (51%) of female patients. The best performing LLM overall was Llama 3.3's 70 billion parameter model using unstructured and structured data with 10-shot chain-of-thought prompts, with a F1 score of 0.894. Unstructured data alone gave the best F1 score for all models except Llama-3.3-70B and Mistral-0.3. Chain-of-thought prompts were more likely to produce better results than basic prompts, with 4 out of the 5 models giving their best performance when prompted by a chain-of-thought prompt. CONCLUSIONS: In this work we demonstrate that open source LLMs can be tailored for use as accurate prehospital pediatric asthma computable phenotypes using EMS data. To our knowledge, this represents the first LLM-based computable phenotypes trained on EMS data.

Ask AI

Helpful

Bookmark