Machine-Learning Derived Individual Risk Prediction in Congenital Heart Surgery: Development and internal validation of risk prediction models for in-hospital mortality after congenital heart surgery using data from the Pediatric Cardiac Critical Care Consortium (PC4) registry

Machine learning (ML) methods may improve prediction of individual risk of mortality risk after congenital heart surgery (CHD) by identifying nonlinear relationships/novel combinations of variables. Improved risk discrimination may improve outcomes by optimizing resource allocation and patient surveillance. Children and adolescents (age≤18 years) undergoing CHD surgery at Pediatric Cardiac Critical Care Consortium hospitals 9/1/2014-12/30/2019 were studied. Non-collinear potential predictors were identified empirically and used to create a logistic regression risk (LR) model (mimicking traditional risk-adjustment models) and ML (random forest and gradient boosting model) models. Model performance was evaluated using net reclassification index (NRI), the proportion of subjects in which either the absolute risk category or estimated risk relative to other subjects changed between each model and one composed only of STAT category as the primary outcome. This metric reflects how often a model changes how a clinician views the risk of a patient relative to a parsimonious model and provides some measure of the utility of these models. Secondary analyses compared overall model fit. In total 51,475 operative encounters from 52 hospitals were studied. Subjects were 45% female, 20% neonates, and 5% underwent STAT 5 operations. C-statistics and Brier scores did not differ significantly between ML and LR models. For absolute risk categorires, point estimates for NRI between ML-models and STAT category (0.76 95%CI: 0.68-0.83) suggested at best modest improvements compared to logistic regression model (0.75 95%CI 0.67-0.83), but the difference was not significant. Similarly, for rankings of risk within the cohort, NRI for ML models (0.26 95%CI: 0.19-0.32) was not significantly different to that for the LR model (0.24 95%CI: 0.17-0.33). ML derived models did not improve overall model fit. Magnitude of improvement in reclassification was at best modest and not statistically significant. Future research is necessary to evaluate the added values of incorporating more comprehensive data and/or advanced computational techniques and use of reclassification in risk prediction.