Machine learning for mortality prediction: do we need yet another model?

We read the study by Arina et al. [1] with interest, especially in the wider context of the peri-operative risk prediction literature. An editorial [2] to accompany a ‘state-of-the-nation’ systematic review of peri-operative risk prediction models made several suggestions for future work in the field: publications should contain in-depth performance statistics; models should be freely available to allow other researchers to validate them externally in new populations; and new models should compare their performance against existing, validated models [3]. We applaud the choice of F1 scores and binary cross-entropy as performance metrics by Arina et al., which have much better properties when event rates are low compared with the traditional discrimination and AUROC metrics. However, we have two concerns about the performance of their model. First, some clarity is needed on when scaling of the continuous variables took place within their analysis workflow. From the manuscript, it is unclear if the scaling was done before splitting the dataset for testing and training. If this were the case, then information about the testing dataset might have leaked back into the training data which could then bias the model and inflate its performance. Our second concern is the complexity of the model. The multi-objective symbolic regression (MOSR) model in the article has more individual predictors than events (85 predictors for 66 deaths) and nearly double the number of regression coefficients when interactions are included (124). The traditional heuristic for logistic regression modelling is that 10 events-per-predictor are needed to prevent model overfitting. While there is debate over such strict events-per-predictors, guidance can be relaxed for certain machine learning algorithms. A discussion justifying the robustness of the MOSR technique with such low events-per-predictors is needed. Though the authors include model equations in an online appendix, we cannot find many of the predictors that are discussed within the article in the regression equations supplied in that appendix; see online Supporting Information Table S1 for details. Body mass index, which is discussed at length, does not seem to be in the published model equation. The authors have not compared this model with existing models predicting the same outcome. The article notes that “few studies predict 1-to 5-year mortality risk in non-cardiac surgery”, but an important one that does is the NZRISK model which is an extension of the SORT model recommended for use in UK patients [4]. The NZRISK model has excellent performance statistics, its regression coefficients are freely available and it has one major benefit over this MOSR, that of parsimony (it requires only eight predictors, all of which seem to be available in this study). Externally validating NZRISK and recalibrating it for a UK population would be a valuable contribution to the literature; it could then serve as a baseline for considering whether more variables (including cardiopulmonary exercise testing) are needed to accurately predict mortality at 1 year post-surgery. We agree with Arina et al. that peri-operative risk prediction needs to move beyond 30 days and look at longer term outcomes. Machine learning approaches often have excellent predictive performance but at the cost of requiring large amounts of data and difficulties interpreting the outputs, which then limits their robustness and clinical utility [5]. We feel that using complex techniques should be justified by showing that they offer significant improvements over existing methods and models. Table S1. Model features: summary of included and missing covariates. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.