Key result
Framingham CHD risk validations fail to update poorly performing rules in ~68% of studies.
Why the study?
External validation studies assess generalizability and contribute to meta-analyses, but it was unknown whether repeated external validation studies of the Framingham Wilson coronary heart disease risk rule generated evidence to their full potential.
Systematic Review (n=40)
Most external validation studies of the Framingham Wilson CHD risk rule fail to generate evidence to their full potential due to lack of updating and incomplete reporting of performance measures.
Hinders meta-analytic synthesis of CHD risk models; challenges researchers to prioritize updating and complete reporting in validations.
BACKGROUND: External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential. METHODS: A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule. RESULTS: We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data. DISCUSSION: Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.
No takes yet. Share an insight, caveat, or question.
Ban et al. (2024) conducted a systematic review in Coronary heart disease risk prediction (n=40). Framingham Wilson CHD risk rule external validation was evaluated on Updating the rule when performing poorly and contribution to meta-analyses of P/O ratio and c statistic. Most external validation studies of the Framingham Wilson CHD risk rule failed to update the rule when it performed poorly (67.5%) or report key performance measures for meta-analysis (56.5%).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: