Answers are grounded in this paper and cite their sources.
Background: Pulmonary hypertension (PH) is a morbid but underrecognized disease diagnosed by right heart catheterization (RHC). Automated extraction of RHC data from electronic health systems is likely to expedite PH diagnosis. However, RHC outputs are typically unstructured, free-text reports and current large language models (LLM) that extract patient-level RHC data are imprecise and derived from small datasets. Methods and Results: We established an iterative development and validation framework using the LLM GPT-4-0125-preview to extract patient-level data from RHC reports. The extraction accuracy was assessed against manual expert-curated ground truth using micro-averaged precision, recall, and F1 metrics across all extracted variables. Evaluation on an independent test set of 100 previously unseen reports yielded a precision of 99.8%, a recall of 99.5%, an F1 score of 99.8%, and a hallucination rate of 0.09%. Next, we deployed the LLM to all n = 17,151 RHC reports captured in the Electronic Health Record of the University of Maryland Medical System (UMMS; ~2.3 million patients across 12 hospitals) from 1 January 2016 to 31 December 2024. Manual validation on 200 random reports demonstrated a precision of 99.7%, a recall of 99.2%, an F1 score of 99.5%, and a hallucination rate of 0.17% (all realistic and physiologically plausible). To further explore reproducibility across deployment settings, we deployed the LLM to a targeted sample of 3000 randomly selected RHC reports that included previously identified error cases. Validation of 200 random reports demonstrated a hallucination rate of 0.33% (precision 99.4%, recall 99.9%, F1 99.6%). Notably, four of the seven observed hallucinations in the large-scale deployment were absent when processed in a smaller deployment cohort. Conclusions: We demonstrate high precision for LLMs when deployed on a scale to RHC reports from a large, diverse multi-hospital EHR. Hallucinations were uncommon overall, but occasional report-level extraction errors persisted despite prompt refinement, highlighting the importance of continued validation during large-scale deployment.
No takes yet. Share an insight, caveat, or question.
Yin et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: