Key result
Combining NLP with PE discharge codes boosts detection F1 scores to ~94% vs NLP alone.
Why the study?
Rule-based NLP tools can identify pulmonary embolism via radiology reports but have limited accuracy and uncertain external validity when used alone.
Does integrating administrative claims data with NLP models improve the accuracy of pulmonary embolism detection in hospitalized patients?
Cross-Sectional (n=381,642)
Yes
Does integrating administrative claims data with NLP models improve the accuracy of pulmonary embolism detection in hospitalized patients?
Integrating rule-based NLP models with administrative claims data significantly improves the accuracy of pulmonary embolism detection compared to using NLP on radiology reports alone.
Answers are grounded in this paper and cite their sources.
May enhance automated PE identification in records; leaves open whether it improves outcomes or trial efficiency.
Background: Rule-based natural language processing (NLP) tools are easy to implement and can identify pulmonary embolism (PE) via radiology reports, but their accuracy is limited when used in isolation, and their external validity remains uncertain. Methods: In this cross-sectional study, we analyzed data from a prespecified sample of 1,712 hospitalized patients (with and without PE) at Mass General Brigham (MGB) hospitals (2016–2021) and applied two previously published NLP algorithms (Verma et al. and Johnson et al) to radiology reports to identify PE. Chart review by two independent physicians using pre-specified criteria was the reference standard. We tested three approaches: (A) NLP applied to all patients; (B) NLP limited to patients with primary or secondary International Classification of Diseases (ICD)-10 PE discharge codes; and (C) NLP applied to patients with PE discharge codes or a Present-on-Admission (POA) indicator (“Y” or “N”) for PE. All others were assumed PE-negative in Approaches B and C to minimize false positives with NLP. Weighted estimates were derived from the full MGB hospitalized cohort (n=381,642) to calculate F1 scores that summarize model performance by combining sensitivity and positive predictive value (PPV) [F1 = 2 x (PPV x sensitivity)/ (PPV + sensitivity)]. Results: In total, 7,708 (2.0%) patients had PE. In Approach A, both NLP models showed high sensitivity (82.5%, 93.0%) and specificity (98.9%, 98.7%) but low PPV (60.3%, 59.6%) (Figure). Approach B improved PPV (95.2%, 94.9%) at the cost of reduced sensitivity (74.1%, 76.2%), while Approach C preserved both high sensitivity (82.5%, 93.0%) and PPV (95.6%, 95.8%). Approach C demonstrated the best performance, yielding significantly higher F1 scores for both NLP models (88.6%, 94.4%) compared with Approach A (69.7%, 72.6%) and Approach B (83.3%, 84.5%) (P<0.001). Conclusions: The accuracy of PE detection improves when rule-based NLP models are operationalized within a screening framework using administrative claims data in addition to radiology reports.
No takes yet. Share an insight, caveat, or question.
Rashedi et al. (2025) conducted a cross-sectional in Pulmonary embolism (n=381,642). NLP integrated with claims data (discharge codes or POA indicator) vs. NLP applied in isolation was evaluated on F1 scores summarizing model performance (p=<0.001). Applying NLP models to patients with PE discharge codes or a Present-on-Admission indicator improved F1 scores for PE detection compared to NLP alone.
Synapse has enriched 10 related papers. Consider them for comparative context: