What drives performance in machine learning models for predicting heart failure outcome?

Figure 1. The suggested pipeline of the predicting process.

Figure 2. Prediction of the models based on the set of clinical covariates utilized. Area under receiver operator curve of the models used splitted by clinical priority of the variables. The confidence interval in this graph are the prediction interval for the area under receiver operator curve prediction. LR, logistic regression; NN, neuron network; RF, random forest; XgBoost, extreme gradient boosting trees; Cox, Cox regression; Majority Ensemble, ensemble model using LR-L1, Cox, XgBoost and NN.

Figure 3. Calibration plot. All models applied were reasonably calibrated (upper panel). Risk distribution of the patients evaluated using the prediction models delineates that most patients had a medium risk for the unified outcome—higher than that assessed for the 1 year mortality outcome (lower panel). The number in parentheses after each model indicates the brier score of each model. The ‘full’ prefix for each model is to indicate that it uses all the 300 + covariates. LR, logistic regression; NN, neuron network; RF, random forest; XGboost, extreme gradient boosting trees; Cox, Cox regression; Majority Ensemble, ensemble model using LR-L1, Cox, XGboost and NN.

Figure 4. Net benefit of the prediction models. Net benefit allows to evaluate the clinical implication of the model in predicting mortality outcome. A threshold of 20% risk for mortality is considered significant for altering decisions regarding treatment strategies and patient’s categorization (Stage C vs. Stage D heart failure). All models give a significant net benefit with random forest being the least beneficial. The straight yellow line denotes the benefit of a model that predicts mortality for all patients. LR, logistic regression; NN, neuron network; RF, random forest; XGboost, extreme gradient boosting trees; Cox, Cox regression; Ensemble, ensemble model using LR-L1, Cox, xgboost and NN.

Figure 5. Area under receiver operator curve plot by types. The relative effect of the various variable types on the performance of the prediction model is delineated. Variable types that are not shown in the graph had a minor (or negative) contribution to the model’s performance. Sex and age contribution are represented by the dashed vertical line. Admin, administrative variables; lab-tests, laboratory tests; physical, physical examination; personal, demographic variables; drug, drug treatment; disease, comorbidities; LR, logistic regression; NN, neuron network; RF, random forest; XGboost, extreme gradient boosting trees; Cox, Cox regression; Majority Ensemble, ensemble model using LR-L1, cox, XGboost and NN.