What question did this study set out to answer?

The study aims to identify the use of natural language processing and machine learning in detecting cardiovascular diagnostic errors from patient safety event reports.

April 12, 2026Open Access

Applying machine learning and natural language processing to patient safety event reports: Identifying patterns of cardiovascular diagnostic errors

Key Points

The study aims to identify the use of natural language processing and machine learning in detecting cardiovascular diagnostic errors from patient safety event reports.
Analyzed patient safety event reports from January 2016 to August 2021.
Manually reviewed reports to label narratives related to cardiovascular diagnostic errors.
Employed four binary machine learning models: logistic regression, elastic net, XGBoost, and deep neural networks.
Evaluated model performance metrics such as AUROC, specificity, and accuracy.
XGBoost identified cardiovascular diagnostic errors with an AUROC of 0.914 and high specificity at 0.982.
The term 'pacemaker' was significant in labeling error-related reports.
Ordering MRI for patients with pacemakers frequently appeared in error narratives.
Key features included 'order', 'EKG', 'cardiac', and 'chest' in identifying diagnostic errors.

Abstract

Objective To explore the utility of natural language processing (NLP) and machine learning (ML) techniques to identify unsafe conditions leading to cardiovascular diagnostic errors using patient safety event (PSE) reports data. Methods PSE reports from January 2016 to August 2021 from a multi-hospital healthcare system in the mid-Atlantic region of the United States were included in this study. To have the true cardiovascular diagnostic errors labels for PSE reports, each individual PSE report was manually reviewed to find clinician-reported narratives describing current definitions of cardiovascular diagnostic errors. The PSE reports which contained cardiovascular diagnostic errors-related narratives were labeled as one and zero otherwise. Four binary ML models were employed to identify cardiovascular diagnostic errors narratives and common features from annotated PSE reports data: (1) simple logistic regression, (2) elastic net, (3) XGBoost, and (4) deep neural networks. Results XGBoost outperformed the rest of the models in identifying cardiovascular diagnostic errors -related reports and achieved high performance metrics on the testing data (AUROC = 0.914, specificity = 0.982, PPV = 0.866, accuracy = 0.929, F-1 score = 0.738, and AUPRC = 0.783). Pacemaker emerged as a significant signal for cardiovascular diagnostic errors in our PSE reports. Our analysis demonstrated that ordering MRI for patients with pacemaker was a frequent theme among cardiovascular diagnostic errors-related PSE reports containing the word pacemaker. Order, EKG, cardiac, and chest were the five most important features in identifying cardiovascular diagnostic errors events. These words were utilized in explaining heart conditions, associated care process, and related safety incidents. Conclusions Findings from our study demonstrates the feasibility of ML and NLP techniques in identifying cardiovascular diagnostic errors-related reports among the existing PSE data sources. However, validation in external healthcare systems is needed before broader application.

Applying machine learning and natural language processing to patient safety event reports: Identifying patterns of cardiovascular diagnostic errors

Key Points

Abstract

Cite This Study