Retrospective comparison of traditional and artificial intelligence-based heart failure phenotyping in a US health system to enable real-world evidence

Objective Quantitatively evaluate the quality of data underlying real-world evidence (RWE) in heart failure (HF). Design Retrospective comparison of accuracy in identifying patients with HF and phenotypic information was made using traditional (ie, structured query language applied to structured electronic health record (EHR) data) and advanced (ie, artificial intelligence (AI) applied to unstructured EHR data) RWE approaches. The performance of each approach was measured by the harmonic mean of precision and recall (F 1 score) using manual annotation of medical records as a reference standard. Setting EHR data from a large academic healthcare system in North America between 2015 and 2019, with an expected catchment of approximately 5 00 000 patients. Population 4288 encounters for 1155 patients aged 18–85 years, with 472 patients identified as having HF. Outcome measures HF and associated concepts, such as comorbidities, left ventricular ejection fraction, and selected medications. Results The average F 1 scores across 19 HF-specific concepts were 49.0% and 94.1% for the traditional and advanced approaches, respectively (p<0.001 for all concepts with available data). The absolute difference in F 1 score between approaches was 45.1% (98.1% relative increase in F 1 score using the advanced approach). The advanced approach achieved superior F 1 scores for HF presence, phenotype and associated comorbidities. Some phenotypes, such as HF with preserved ejection fraction, revealed dramatic differences in extraction accuracy based on technology applied, with a 4.9% F 1 score when using natural language processing (NLP) alone and a 91.0% F 1 score when using NLP plus AI-based inference. Conclusions A traditional RWE generation approach resulted in low data quality in patients with HF. While an advanced approach demonstrated high accuracy, the results varied dramatically based on extraction techniques. For future studies, advanced approaches and accuracy measurement may be required to ensure data are fit-for-purpose.