47

Introduction: The National Quality Forum has endorsed the use of a modified version of the Mortality Probability Model (MPM0-III) for comparing observed and predicted hospital mortality in U.S. ICUs. However, recent studies suggest that this model is less accurate than current versions of other prognostic models. Hypothesis: That the performance of MPM0-III and APACHE IV for predicting hospital mortality would not differ when tested concurrently in a large multi-institutional ICU database. Methods: Retrospective cohort study using day 1 information from 51,825 first admissions in 46 ICUs at 34 U.S. hospitals during 1/1/2008 to 6/30/2012. For each MPM0-III-eligible patient we calculated the probability of hospital mortality using the MPM0-III and APACHE IV predictive models. We compared each model’s prediction against actual mortality using the following measures of accuracy: discrimination was assessed by the area under the receiver operating characteristic curve (AUROC); calibration was assessed by the standardized mortality ratio (SMR), Hosmer-Lemeshow (H-L) statistic, and a modified Brier score. The latter measure consisted of determining the Brier score resulting from using the observed mortality as a constant prediction, and then calculating the percentage reduction the actual Brier score represented (higher percentage = better accuracy). Results: The observed hospital mortality rate was 11.2%, while the APACHE IV predicted mortality was 12.6% (SMR = 0.89), and MPM0-III predicted mortality was 15.2% (SMR = 0.74). For APACHE IV the AUROC was 0.884 and H-L statistic was 12.4. For MPM0-III the AUROC was 0.838 and H-L statistic 109.8. Differences between the two models for the AUROC and SMR, respectively were highly significant (p<0.001). The percent reduction in prediction error from the null model shown by the Brier score was 31.4% for APACHE IV and 18.5% for MPM0-III. Conclusions: In a side-by-side comparison within a large dataset, the APACHE IV prognostic model had better discrimination and calibration than the MPM0-III model. Healthcare reporting agencies should take this information into account when considering critical care outcome measures.