Key result
AI ECG model occlusion detection drops to an F1 score of ~0.53 after removing theatre noise.
Why the study?
Deep learning has emerged as an effective technique in automated ECG analysis, prompting evaluation of its feasibility for detecting induced coronary artery occlusion earlier than experienced cardiologists.
Can a deep learning algorithm applied to 9-lead ECGs detect induced acute complete coronary artery occlusion earlier and more accurately than experienced cardiologists or STEMI criteria, and how do confounding data features affect its performance?
Observational (n=104)
No
Can a deep learning algorithm applied to 9-lead ECGs detect induced acute complete coronary artery occlusion earlier and more accurately than experienced cardiologists or STEMI criteria, and how do confounding data features affect its performance?
This study demonstrates that deep learning ECG models can exploit unmeasured confounders such as catheterization laboratory electrical noise (data leakage) to produce spuriously high diagnostic accuracy for acute coronary occlusion that vanishes when baseline recording environments are standardized.
Answers are grounded in this paper and cite their sources.
Deep learning ECG models risk inflated accuracy from environmental data leakage; challenges prior validity claims and leaves open reliable coronary occlusion detection after standardization.
Aims: Deep learning (DL) has emerged in recent years as an effective technique in automated ECG analysis. Methods and results: A retrospective, observational study was designed to assess the feasibility of detecting induced coronary artery occlusion in human subjects earlier than experienced cardiologists using a DL algorithm. A deep convolutional neural network was trained using data from the STAFF III database. The task was to classify ECG samples as showing acute coronary artery occlusion, or no occlusion. Occluded samples were recorded after 60 s of balloon occlusion of a single coronary artery. For the first iteration of the experiment, non-occluded samples were taken from ECGs recorded in a restroom prior to entering theatres. For the second iteration of the experiment, non-occluded samples were taken in the theatre prior to balloon inflation. Results were obtained using a cross-validation approach. In the first iteration of the experiment, the DL model achieved an F1 score of 0.814, which was higher than any of three reviewing cardiologists or STEMI criteria. In the second iteration of the experiment, the DL model achieved an F1 score of 0.533, which is akin to the performance of a random chance classifier. Conclusion: The dataset was too small for the second model to achieve meaningful performance, despite the use of transfer learning. However, 'data leakage' during the first iteration of the experiment led to falsely high results. This study highlights the risk of DL models leveraging data leaks to produce spurious results.
No takes yet. Share an insight, caveat, or question.
Brisk et al. (2021) conducted an observational in Induced acute coronary artery occlusion (n=104). Deep learning algorithm (34-layer convolutional neural network) for ECG classification vs. Consensus of three consultant cardiologists and STEMI criteria was evaluated on Classification of acute coronary artery occlusion vs no occlusion on ECG (measured by F1 score and accuracy) (95% CI 0.433-0.633). A deep learning ECG model achieved an F1 score of 0.814 with confounded baseline data, which dropped to 0.533 after eliminating theatre noise data leakage.
Synapse has enriched 10 related papers. Consider them for comparative context: