Why the study?
Risk prediction models commonly inform treatment decisions, but uncertainty around risk scores beyond confidence intervals is rarely explored.
Does the choice of modelling decisions affect individual cardiovascular risk predictions in patients eligible for primary prevention?
Does the choice of modelling decisions affect individual cardiovascular risk predictions in patients eligible for primary prevention?
Risk prediction models using routinely collected data show large variability in individual patient risk depending on modelling decisions, which could reclassify millions of patients for statin eligibility despite performing similarly on standard metrics.
Answers are grounded in this paper and cite their sources.
Wide variation in individual risk estimates across models; leaves open whether standardized approaches are needed for primary prevention decisions.
BACKGROUND: Risk prediction models are commonly used in practice to inform decisions on patients' treatment. Uncertainty around risk scores beyond the confidence interval is rarely explored. We conducted an uncertainty analysis of the QRISK prediction tool to evaluate the robustness of individual risk predictions with varying modelling decisions. METHODS: We derived a cohort of patients eligible for cardiovascular risk prediction from the Clinical Practice Research Datalink (CPRD) with linked hospitalisation and mortality records (N = 3,792,474). Risk prediction models were developed using the methods reported for QRISK2 and 3, before adjusting for additional risk factors, a secular trend, geographical variation in risk and the method for imputing missing data when generating a risk score (model A-model F). Ten-year risk scores were compared across the different models alongside model performance metrics. RESULTS: We found substantial variation in risk on the individual level across the models. The 95 percentile range of risks in model F for patients with risks between 9 and 10% according to model A was 4.4-16.3% and 4.6-15.8% for females and males respectively. Despite this, the models were difficult to distinguish using common performance metrics (Harrell's C ranged from 0.86 to 0.87). The largest contributing factor to variation in risk was adjusting for a secular trend (HR per calendar year, 0.96 [0.95-0.96] and 0.96 [0.96-0.96]). When extrapolating to the UK population, we found that 3.8 million patients may be reclassified as eligible for statin prescription depending on the model used. A key limitation of this study was that we could not assess the variation in risk that may be caused by risk factors missing from the database (such as diet or physical activity). CONCLUSIONS: Risk prediction models that use routinely collected data provide estimates strongly dependent on modelling decisions. Despite this large variability in patient risk, the models appear to perform similarly according to standard performance metrics. Decision-making should be supplemented with clinical judgement and evidence of additional risk factors. The largest source of variability, a secular trend in CVD incidence, can be accounted for and should be explored in more detail.
No takes yet. Share an insight, caveat, or question.
Pate et al. (2019) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: