Comparing machine learning models with established risk scores in predicting bleeding and ischaemic stroke in patients with atrial fibrillation undergoing transcatheter aortic valve implantation

Abstract Background Patients with atrial fibrillation (AF) after successful transcatheter aortic valve implantation (TAVI) are at heightened risk of ischaemic stroke (IS) and bleeding. However, risk scores, such as CHA2DS2-VA and HAS-BLED, provide modest prediction of IS and bleeding. Although traditional statistical methods (e.g., Cox regression) allow for identifying patients at higher risk for these events, a machine learning (ML) approach may enhance risk prediction by capturing complex non-linear associations. Purpose To develop and evaluate ML models predicting clinical events in patients with AF after TAVI with established risk scores for IS and bleeding. Methods Patient-level data from the ENVISAGE-TAVI AF trial were used to develop ML models for the prediction of IS, major gastrointestinal bleeding (MGIB), all clinically relevant bleeding (major or clinically relevant nonmajor bleeding), and net adverse clinical events (NACE; composite of death from any cause, myocardial infarction, IS, systemic thromboembolic event, valve thrombosis, or major bleeding). For each outcome, 10 ML algorithms were trained, optimised, and ranked by performance using nested cross-validation. The model with the highest F1 score (harmonic mean of precision and recall) for each outcome was selected and validated on a separate hold-out set (25%). SHAP (SHapley Additive exPlanations) values were calculated to determine the average magnitude of feature contributions. Using F1 scores, the best model of each outcome was compared with logistic regression models trained exclusively on CHA2DS2-VA (IS) and HAS-BLED (bleeding) scores. Results Of 1377 patients on treatment, 41 had IS, 83 had MGIB, 375 had clinically relevant bleeding, and 255 had NACE. The predictive abilities of a linear discriminant analysis algorithm for IS (F1 score=0.08) and CHA2DS2-VA (F1 score=0.09) were similarly low and numerically better than HAS-BLED (F1 score=0.05; Figure 1). Prediction of MGIB was similarly low for a logistic-lasso algorithm (F1 score=0.11), CHA2DS2-VA (F1 score=0.09), and HAS-BLED (F1 score=0.12). For all clinically relevant bleeding, the predictive performance of a naïve-Bayes algorithm (F1 score=0.39) was similar to that of CHA2DS2-VA (F1 score=0.38) and HAS-BLED (F1 score=0.41). The predictive ability of a logistic regression algorithm for NACE (F1 score=0.33) was numerically better than CHA2DS2-VA (F1 score=0.22) or HAS-BLED (F1 score=0.27). Low event rates were generally observed to limit the predictive power of ML models and scores. All 4 algorithms identified novel predictors of events (Figure 2). Conclusion ML models allow for risk assessment for IS, MGIB, all clinically relevant bleeding, and NACE, and were comparable to traditional risk scores. While further development and validation in larger datasets will be required, these initial models reveal potential new predictors of IS and bleeding events that may be important to consider in future studies.