Key result
Customization of the APACHE III scoring system improved overall calibration for hospital mortality prediction (SMR 0.94) compared to the original model (SMR 1.18), but failed to achieve uniform goodness-of-fit across patient subgroups.
Why the study?
Mortality predictions using APACHE II and III scoring scales are often inaccurate in populations different from those in which they were developed due to case-mix differences.
Does customization of APACHE II and III scores improve mortality prediction in an interdisciplinary ICU population?
Cohort (n=2,795)
No
Does customization of APACHE II and III scores improve mortality prediction in an interdisciplinary ICU population?
Effect estimate: SMR 0.94 (95% CI 0.83-1.06)
Absolute Event Rate: 0.94% vs 1.18%
Customization of APACHE III improves overall calibration but fails to achieve uniform fit across subgroups, offering no advantage for quality of care comparisons.
BACKGROUND: Mortality predictions calculated using scoring scales are often not accurate in populations other than those in which the scales were developed because of differences in case-mix. The present study investigates the effect of first-level customization, using a logistic regression technique, on discrimination and calibration of the Acute Physiology and Chronic Health Evaluation (APACHE) II and III scales. METHOD: Probabilities of hospital death for patients were estimated by applying APACHE II and III and comparing these with observed outcomes. Using the split sample technique, a customized model to predict outcome was developed by logistic regression. The overall goodness-of-fit of the original and the customized models was assessed. RESULTS: Of 3383 consecutive intensive care unit (ICU) admissions over 3 years, 2795 patients could be analyzed, and were split randomly into development and validation samples. The discriminative powers of APACHE II and III were unchanged by customization (areas under the receiver operating characteristic [ROC] curve 0.82 and 0.85, respectively). Hosmer-Lemeshow goodness-of-fit tests showed good calibration for APACHE II, but insufficient calibration for APACHE III. Customization improved calibration for both models, with a good fit for APACHE III as well. However, fit was different for various subgroups. CONCLUSIONS: The overall goodness-of-fit of APACHE III mortality prediction was improved significantly by customization, but uniformity of fit in different subgroups was not achieved. Therefore, application of the customized model provides no advantage, because differences in case-mix still limit comparisons of quality of care.
No takes yet. Share an insight, caveat, or question.
Markgraf et al. (2001) conducted a cohort in Critical illness requiring ICU admission (n=2,795). Customization of APACHE II and III scoring systems vs. Original APACHE II and III scoring systems was evaluated on Hospital mortality prediction calibration (Standardized Mortality Ratio) in the validation sample for APACHE III (SMR 0.94, 95% CI 0.83-1.06). Customization of the APACHE III scoring system improved overall calibration for hospital mortality prediction (SMR 0.94) compared to the original model (SMR 1.18), but failed to achieve uniform goodness-of-fit across patient subgroups.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: