Key result
External validation of ACS clinical prediction models shows a ~24% drop in discrimination, risking harm.
Why the study?
Clinical prediction models often do not perform as expected when tested on new databases, and independent external validations are recommended but often not performed.
Does recalibration of published clinical prediction models for acute coronary syndromes improve their performance and clinical utility when applied to new populations?
Systematic Review
Does recalibration of published clinical prediction models for acute coronary syndromes improve their performance and clinical utility when applied to new populations?
Absolute Event Rate: 0.7% vs 0.76%
Applying published clinical prediction models for ACS 'off-the-shelf' to new populations often risks net harm due to poor calibration, highlighting the critical need for model recalibration before clinical use.
Unrecalibrated ACS prediction models risk net harm in new populations; confirms mandatory local recalibration before clinical or research use.
Purpose It is increasingly recognized that clinical prediction models (CPMs) often do not perform as expected when they are tested on new databases. Independent external validations of CPMs are recommended but often not performed. Here we conduct independent external validations of acute coronary syndrome (ACS) CPMs. Methods A systematic review identified CPMs predicting outcomes for patients with ACS. Independent external validations were performed by evaluating model performance using individual patient data from 5 large clinical trials. CPM performance with and without various recalibration techniques was evaluated with a focus on CPM discrimination (c-statistic, % relative change in c-statistic) as well as calibration (Harrell’s E avg , E 90 , Net Benefit). Results Of 269 ACS CPMs screened, 23 (8.5%) were compatible with at least one of the trials and 28 clinically appropriate external validations were performed. The median c statistic of the CPMs in the derivation cohorts was 0.76 (IQR, 0.74 to 0.78). The median c-statistic in these external validations was 0.70 (IQR, 0.66 to 0.71) reflecting a 24% decrement in discrimination. However, this decrement in discrimination was due mostly to narrower case-mix in the validation cohorts compared to derivation cohorts, as reflected in the median model based c-statistic [0.71 (IQR 0.66 to 0.75). The median calibration slope in external validations was 0.84 (IQR, 0.72 to 0.98) and the median E avg (standardized by the outcome rate) was 0.4 (IQR, 0.3 to 0.8). Net benefit indicates that most CPMs had a high risk of causing net harm when not recalibrated, particularly for decision thresholds not near the overall outcome rate. Conclusion Independent external validations of published ACS CPMs demonstrate that models tested in our sample had relatively well-preserved discrimination but poor calibration when externally validated. Applying ‘off-the-shelf’ CPMs often risks net harm unless models are recalibrated to the populations on which they are used.
No takes yet. Share an insight, caveat, or question.
Wessler et al. (2021) conducted a systematic review in Acute Coronary Syndromes (ACS). Clinical prediction models (CPMs) vs. Derivation cohorts was evaluated on CPM discrimination (c-statistic). Independent external validations of acute coronary syndrome clinical prediction models showed a 24% decrement in discrimination and poor calibration, risking net harm without recalibration.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: