Abstract Background Recognizing patient symptoms is fundamental to medicine, research, and public health. However, symptoms are often underreported in coded formats even though they are routinely documented in physician notes. Large language models (LLMs), noted for their generalizability, could help bridge this gap by mimicking the role of human expert chart reviewers for symptom identification. Objective The primary objective of this multisite study was to measure the accurate identification of infectious respiratory disease symptoms using LLMs instructed to follow chart review guidelines. The secondary objective was to evaluate LLM generalizability in multisite settings without the need for site-specific training, fine-tuning, or customization. Methods Four LLMs were evaluated: GPT-4, GPT-3.5, Llama2 70B, and Mixtral 8×7B. LLM prompts were instructed to take on the role of chart reviewers and follow symptom annotation guidelines when assessing physician notes. Ground truth labels for each note were annotated by subject matter experts. Optimal LLM prompting strategies were selected using a development corpus of 103 notes from the emergency department at Boston Children’s Hospital. The performance of each LLM was measured using a test corpus with 202 notes from Boston Children’s Hospital. The performance of an International Classification of Diseases, Tenth Revision ( ICD-10 )–based method was also measured as a baseline. Generalizability of the most performant LLM was then measured in a validation corpus of 308 notes from 21 emergency departments in the Indiana Health Information Exchange. Results Symptom identification accuracy was superior for every LLM tested for each infectious disease symptom compared to an ICD-10 –based method ( F 1 -score=45.1%). GPT-4 was the highest scoring ( F 1 -score=91.4%; P <.001) and was significantly better than the ICD-10 –based method, followed by GPT-3.5 ( F 1 -score=90.0%; P <.001), Llama2 ( F 1 -score=81.7%; P <.001), and Mixtral ( F 1 -score=83.5%; P <.001). For the validation corpus, performance of the ICD-10 –based method decreased ( F 1 -score=26.9%), while GPT-4 increased ( F 1 -score=94.0%), demonstrating better generalizability using GPT-4 ( P <.001). Conclusions LLMs significantly outperformed an ICD-10 –based method for respiratory symptom identification in emergency department electronic health records. GPT-4 demonstrated the highest accuracy and generalizability, suggesting that LLMs may augment or replace traditional approaches. LLMs can be instructed to mimic human chart reviewers with high accuracy. Future work should assess broader symptom types and health care settings.
McMurry et al. (Thu,) studied this question.