December 9, 2009Epidemiology5,085 citationsOpen Access
Assessing the Performance of Prediction Models
View Full Paper
Key Points
- To define the role and utility of traditional, novel, and decision-analytic metrics in assessing the performance of clinical prediction models.
- Evaluated traditional performance measures (Brier score, concordance statistic/ROC curve, goodness-of-fit calibration) alongside newer metrics (survival c-statistics, net reclassification improvement [NRI], integrated discrimination improvement [IDI], and decision curves).
- Demonstrated performance metric applications using a testicular cancer case study predicting residual tumor versus benign tissue (n=544 for model development, n=273 for external validation).
- Assessing and reporting both discrimination (distinguishing between outcomes) and calibration (agreement between predicted and observed risks) remain essential requirements for every prediction model.
- Decision-analytic measures, including decision curves assessing net benefit, are necessary whenever models are intended to guide clinical decision-making.
- Novel reclassification metrics such as NRI and IDI offer specific value when evaluating the incremental predictive gain of adding new biomarkers or variables to established models.
Abstract
The performance of prediction models can be assessed using a variety of methods and metrics. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance (or c) statistic for discriminative ability (or area under the receiver operating characteristic ROC curve), and goodness-of-fit statistics for calibration.Several new measures have recently been proposed that can be seen as refinements of discrimination measures, including variants of the c statistic for survival, reclassification tables, net reclassification improvement (NRI), and integrated discrimination improvement (IDI). Moreover, decision-analytic measures have been proposed, including decision curves to plot the net benefit achieved by making decisions based on model predictions.We aimed to define the role of these relatively novel approaches in the evaluation of the performance of prediction models. For illustration, we present a case study of predicting the presence of residual tumor versus benign tissue in patients with testicular cancer (n = 544 for model development, n = 273 for external validation).We suggest that reporting discrimination and calibration will always be important for a prediction model. Decision-analytic measures should be reported if the predictive model is to be used for clinical decisions. Other measures of performance may be warranted in specific applications, such as reclassification metrics to gain insight into the value of adding a novel predictor to an established model.