Enhancing Vaccine Safety Surveillance: Extracting Vaccine Mentions from Emergency Department Triage Notes Using Fine-Tuned Large Language Models

Key Points

The fine-tuned Llama 3 billion parameter model showed superior accuracy in extracting vaccine mentions from triage notes.
Prompt engineering helped create a labeled dataset, which was validated by human annotators before model testing.
Comparative analysis included prompt-engineered models, fine-tuned models, and a rule-based approach for data extraction.
Results highlight the capability of large language models to support vaccine safety surveillance and detect adverse events efficiently.

Abstract

This study evaluates fine-tuned Llama 3.2 models for extracting vaccine-related information from emergency department triage notes to support near real-time vaccine safety surveillance. Prompt engineering was used to initially create a labeled dataset, which was then confirmed by human annotators. The performance of prompt-engineered models, fine-tuned models, and a rule-based approach was compared. The fine-tuned Llama 3 billion parameter model outperformed other models in its accuracy of extracting vaccine names. Model quantization enabled efficient deployment in resource-constrained environments. Findings demonstrate the potential of large language models in automating data extraction from emergency department notes, supporting efficient vaccine safety surveillance and early detection of emerging adverse events following immunization issues.

Enhancing Vaccine Safety Surveillance: Extracting Vaccine Mentions from Emergency Department Triage Notes Using Fine-Tuned Large Language Models

Key Points

Abstract

Cite This Study