Key result
External validation of cardiovascular clinical prediction models showed a median percent decrease in discrimination of -11.1% compared to derivation data, with 81% of validations reporting lower discrimination.
Why the study?
Although many clinical prediction models exist for cardiovascular disease, the extent to which they have undergone external validation and how well they generally perform had not been broadly evaluated.
Systematic Review (n=2,030)
Effect estimate: -11.1% (95% CI -32.4 to 2.7)
Many cardiovascular clinical prediction models lack external validation, and those that are validated often show overly optimistic performance in their derivation datasets.
Many cardiovascular CPMs lack robust external validation with heterogeneous performance; leaves open their reliability for guiding treatment decisions.
Background There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested and how well they generally perform has not been broadly evaluated. Methods A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts PACE CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results 2030 external validations of 1382 CPMs were identified. 807 (58%) of the CPMs in the Registry have never been externally validated. On average there were 1.5 validations per CPM (range 0-94). The median external validation AUC was 0.73 (25 th −75 th percentile [IQR] 0.66, 0.79), representing a median percent decrease in discrimination of −11.1% (IQR −32.4%, +2.7%) compared to performance on derivation data. 81% (n = 1333) of validations reporting AUC showed discrimination below that reported in the derivation dataset. 53% (n = 983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was −3.7% (IQR −13.2, 3.1) for ‘closely related’ validations (n=123), −9.0 (IQR −27.6, 3.9) for ‘related validations’ (n=862) and −17.2% (IQR −42.3, 0) for ‘distantly related’ validations (n=717) (p<0.001). Conclusion Many published cardiovascular CPMs have never been externally validated and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
No takes yet. Share an insight, caveat, or question.
Wessler et al. (2021) conducted a systematic review in Cardiovascular disease (n=2,030). External validation of clinical prediction models vs. Derivation dataset performance was evaluated on Median percent change in discrimination (AUC) compared to derivation data (-11.1%, 95% CI -32.4 to 2.7). External validation of cardiovascular clinical prediction models showed a median percent decrease in discrimination of -11.1% compared to derivation data, with 81% of validations reporting lower discrimination.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: