Risk prediction models developed in other countries require validation and recalibration before use in a new country setting to ensure accurate risk-adjusted outcomes.
OBJECTIVE: To assess the performance of published risk prediction models in common use in adult critical care in the United Kingdom and to recalibrate these models in a large representative database of critical care admissions. DESIGN: Prospective cohort study. SETTING: A total of 163 adult general critical care units in England, Wales, and Northern Ireland, during the period of December 1995 to August 2003. PATIENTS: A total of 231,930 admissions, of which 141,106 met inclusion criteria and had sufficient data recorded for all risk prediction models. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The published versions of the Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE II UK, APACHE III, Simplified Acute Physiology Score (SAPS) II, and Mortality Probability Models (MPM) II were evaluated for discrimination and calibration by means of a combination of appropriate statistical measures recommended by an expert steering committee. All models showed good discrimination (the c index varied from 0.803 to 0.832) but imperfect calibration. Recalibration of the models, which was performed by both the Cox method and re-estimating coefficients, led to improved discrimination and calibration, although all models still showed significant departures from perfect calibration. CONCLUSIONS: Risk prediction models developed in another country require validation and recalibration before being used to provide risk-adjusted outcomes within a new country setting. Periodic reassessment is beneficial to ensure calibration is maintained.
Harrison et al. (Thu,) studied this question.