Cardiopulmonary exercise testing before lung resection surgery: still indicated? Evaluating predictive utility using machine learning

Figure 1. Summary of model performance metrics for the prediction of postoperative complications. (A–C) The results for AUC-ROC, AUC-PR and F1-Score, respectively. Each cell contains the mean and SD of the respective metric, computed on the test data from each of the five outer-loop iterations of the nested cross-validation. Unselected refers to all patients in our study. ACCP and ERS/ESTS criteria refer to patients selected according to the respective guidelines.1 2 ACCP, American College of Chest Physicians; AUC-PR, area under the precision-recall curve; AUC-ROC, area under the curve of the receiver operating characteristic; CPET, cardiopulmonary exercise testing; ERS/ESTS, European Respiratory Society/European Society of Thoracic Surgery; LR, logistic regression; NN, neural network; PCCs, postoperative cardiovascular complications; PPCs, postoperative pulmonary complications; RF, random forest; SVM, support vector machine; XGB, XGBoost.

Figure 2. Effect of CPET parameter inclusion on model performance metrics for the prediction of postoperative complications. Each cell of ΔAUC-ROC, ΔAUC-PR and ΔF1-score contains the mean difference between the ‘Standard+CPET’ and ‘Standard’ profiles, along with the corresponding SD, comparing the respective performance metric computed on the test data from each of the five outer-loop iterations of the nested cross-validation. NRI is a comparative metric that quantifies how well a ‘new’ model (in this case, trained on the ‘Standard+CPET’ profile) improves the classification of outcomes (occurrence of complications) compared with an ‘old’ model (trained on the ‘Standard’ profile).31 Here, NRI is the difference in the proportion of correctly vs incorrectly reclassified patients, both among those with postoperative complications and those without. As with the other metrics, NRI is reported with mean and standard deviation (based on the same fivefold nested cross-validation procedure). P values indicate whether the ‘Standard+CPET’ profile resulted in a statistically significant improvement in predictive model performance over the ‘Standard’ profile. Unselected refers to all patients in our…

Figure 3. Contribution of individual variables to the prediction of postoperative complications in logistic regression models. SHAP analysis of logistic regression models retrained to include resting PETCO2. The mean AUC-ROC and corresponding SD are reported. The SHAP value represents the contribution of a variable to an individual prediction and provides insight into its importance. A positive SHAP value means that the value of the variable contributed to predicting the occurrence of a postoperative complication, whereas a negative SHAP value indicates a contribution towards predicting the absence of a postoperative complication. Each dot represents the contribution of a variable to the prediction for a patient. Dots are vertically displaced to show their density. For continuous variables, colour corresponds to the value of the variable with respect to the mean, which is reported alongside the variable name. For binary variables, high means that the respective variable is true and low that it is not. In the case of thoracotomy/VATS, high corresponds to thoracotomy and low to VATS. Only the 15 most important variables (ie, the top half) are included, sorted according to their mean absolute…