ABSTRACT Introduction and Objective Traditional adverse safety events (ASE) identification relies on domain experts to manually review and annotate charts, which hinders the scalability of processing high‐volume EMS data. This study explores the use of large language model (LLM) with a knowledge base to automate extraction of adverse safety events (ASE) from unstructured emergency medical service (EMS) notes for pediatric out‐of‐hospital cardiac arrest (OHCA) as proof of concept. Data Sources and Study Design Pediatric OHCA records from a national EMS provider were obtained from 2017 to 2020. Leveraging the Pediatric Prehospital Adverse Safety Event Detection System (PEDS) as a foundational knowledge base, we used the LinkML framework to develop an ontology to define ASEs across six essential EMS care domains. To convert unstructured EMS narratives into structured prompts, we used the Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES) method, which generated schema‐driven prompts to guide the GPT‐3.5 model in identifying ASEs. By mapping unstructured data into structured concepts consistent with PEDS guidelines, the model produced targeted prompts that supported effective entity extraction. Results We evaluated framework effectiveness with accuracy, recall, precision, F1 score, and specificity across 42 pediatric OHCA cases covering ASE‐related entities. RescueGPT showed high accuracy in detecting common ASEs (Patient Rhythm, Age, Weight, Length) but revealed challenges in rare events (Failure to Establish IV Access, Incorrect Airway Equipment Size, Failure to Ventilate Patient) likely due to more inconsistent and complex documentation. Conclusions RescueGPT demonstrates potential in scaling automated ASE detection, but performance varies by completeness and clarity of EMS narrative, particularly with rare events. Fragmented clinical documentation limits accuracy and highlights the need for standardized collection protocols in EMS systems. Future directions will focus on implementing rebalancing strategies for rare events, applying explainability methods to improve decision‐making transparency, and refining text segmentation techniques to handle mixed outcomes to further improve performance.
Hsieh et al. (Fri,) studied this question.