A68-18 Accuracy of a Deep Learning Model for Radiographic Pneumonia in Urgent Care Clinics

Rationale Community-acquired pneumonia (CAP) is a common outpatient concern; to aid in its detection and management, an electronic clinical decision support tool providing real-time likelihood of pneumonia (ePneumonia) was implemented in 5 Intermountain Health (IH) urgent care clinics. A key component of CAP diagnosis and ePneumonia is the finding of lung opacity on chest imaging. Our group previously developed a deep learning neural network (CheXED) to ascertain radiographic pneumonia in emergency departments and provide data for ePneumonia. While CheXED has been updated for outpatient use, the model’s real-world accuracy has not been assessed. Methods We obtained chest imaging from IH urgent care visits between 4/1/24 and 3/31/25 that were associated with a pneumonia ICD-10 code, resulting in a set of 1011 chest PA and lateral radiographs. CheXED evaluated these images in real-time, providing a result of “positive” or “negative” for radiographic pneumonia. Clinical radiologist interpretations of these chest radiographs were screened to “positive” or “negative” by an internal natural language processing algorithm. Interpretations that were indeterminate or “positive” were then manually categorized. CheXED output was then compared with the radiology interpretations. Of the 1011 images, 33 patients had follow-up chest imaging (radiograph or CT scan) within 3 days. Radiologist interpretation of follow-up imaging was used as a gold standard for the presence of radiographic pneumonia and compared to the CheXED output of initial imaging. Results CheXED findings were consistent with radiologist interpretation in 79% of initial images (n = 802), with 71% “positive” for pneumonia (n = 720) and 8% “negative” for pneumonia (n = 82) (see Table 1). CheXED findings on initial radiographs were consistent with radiologist interpretation of follow-up imaging in 82% of imaging sets (n = 27), with 73% “positive” for pneumonia (n = 24) and 9% “negative” for pneumonia (n = 3). When compared to radiologist interpretation of follow-up imaging, CheXED had 92% sensitivity, 43% specificity, 86% positive predictive value, and 60% negative predictive value. Conclusion CheXED was consistent with radiologist interpretation in 79% of the initial imaging and 82% of the follow-up imaging; this is similar to the interpretation variability rate among board-certified radiologists (Albaum CHEST 1996). Using the radiologist interpretation of follow-up imaging as the gold standard for presence of pneumonia, CheXED had a positive predictive value of 86%. Therefore, a “positive” CheXED result is likely to represent radiographic pneumonia, with a low false positive rate. Overall, CheXED can accurately inform outpatient providers about the presence of radiographic pneumonia.