Highlighting uncertainty in clinical risk prediction using a model of emergency laparotomy mortality risk

Fig. 1. Point estimates of mortality risk contrasted with predicted distributions over mortality risk. The point estimates from conventional models mask whether our prediction is relatively certain (as in Patient 1) or uncertain (as in Patient 3). The median predicted risk is used as the point estimate in each case. Patients 2 and 3 may have a mortality risk much higher than the median, and the point estimates fail to communicate this.

Fig. 2. Changes in predicted mortality risk with each RUNE variable. “Partial dependence” plots in panels a–o show how the mortality risk predicted by RUNE changes with each variable if the other variables are held fixed. Blue and orange bands show changes in predicted mortality risk. In panels, a–l and o, the band’s width for each variable is proportional to the uncertainty of its association with mortality risk. From lightest to darkest, the four overlapping bands represent 95, 70, 45, and 20% confidence. Grey histograms show the distribution of each variable. Base categories are excluded. In panel m, respiratory status is encoded as 0: No dyspnoea, 1: Mild COPD or dyspnoea, 2: Moderate COPD or dyspnoea, 3: Fibrosis or consolidation or severe dyspnoea. Also in panel m, cardiovascular status is encoded as 0: No cardiac failure, 1: Cardiovascular medications, 2: Peripheral oedema or taking warfarin, 3: Raised jugular venous pressure or cardiomegaly.

Fig. 3. Calibration curves. Calibration curves are shown for the re-fitted NELA Calculator (a) and RUNE (b). Each of the 120 curves for each model is derived using one of the 120 validation sets in the cross-validation. Grey histograms show the distribution of each model’s predicted mortality risks (the range of point estimates of risk across all cases) across all validation sets.

Fig. 4. RUNE’s predictions for the case study patient. Lactate and albumin are not measured when the patient is first assessed, and our sub-models, therefore, predict distributions of plausible values for them (a, b). Based upon this, RUNE predicts a wide mortality risk distribution (c, blue), indicating significant uncertainty which is masked by the point estimate from the NELA calculator. Once albumin and lactate are measured and found to be surprisingly deranged, the predicted albumin and lactate are disregarded, and RUNE makes a confident prediction of higher mortality risk (c, orange).