Answers are grounded in this paper and cite their sources.
Hupertan et al.1 have done a nice analysis of the accuracy of a previously published nomogram2 when applied to an external dataset. The authors used a sample of 565 men who were treated at a single institution in France. They found that the performance of the nomogram, as measured by the concordance index (similar to the area under a receiver operating characteristic curve) dropped from 0.74 to 0.607. The authors suggest dynamic revision of renal cell cancer prediction models, although they acknowledge that the current nomogram may still be the best alternative. It would be interesting to see a model-free assessment of the calibration of the nomogram. In Figure 2 of Hupertan et al., a scatter plot is presented that compares nomogram-predicted probabilities with predicted probabilities from a Cox model fit to the French data. The plot suggests that the 2 models' predictions do not agree well; however, it would be of considerably more interest to determine how the nomogram-predicted probabilities agree with actual patient outcome. This could be done by grouping patients with respect to their predicted probabilities then comparing the mean prediction for each group with the actual (eg, Kaplan-Meier) estimate. Such a presentation would avoid the assumption that a Cox model fit to a dataset was a gold standard. Although assessment of the nomogram calibration could be questioned, discrimination of the nomogram (ie, how well it ranks patients) is measured precisely by using the concordance index. In their initial publication, Kattan et al. reported a concordance index of 0.74. However, Hupertan et al. obtained a considerably lower value (0.607). Curiously, in a previous validation study, Cindolo et al. actually obtained a higher value (0.807). It would be convenient if a reliable threshold existed, above which a tool would be deemed clinically useful and below which a tool would be deemed not clinically useful. However, that threshold is remarkably difficult to establish, although it is a subject of active methodological research. Without that threshold, it is hard to know how to react when the concordance index jumps like this. What is the comparison? Because the concordance index will change at least to some degree with each new assessment, a direct comparison of alternatives greatly facilitates decision making. In other words, head-to-head comparison of prediction approaches applied to a common dataset generally yields concordance indices that are straightforward to compare. In that set-up, in which the decision maker is forced to pick among alternative prediction approaches, he or she typically can pick the winner of this contest. This type of comparison was a major strength of the study by Cindolo et al. They compared 4 prognostic models using data from 3 European centers. Those authors found that the model by Kattan et al. consistently was the most accurate, with its concordance index varying from 0.784 to 0.847 across the centers. With this approach, the alternatives are compared directly, without having to judge the concordance index in isolation or against a possibly arbitrary threshold. How should a clinician respond when witnessing a drop in a model's concordance index? From a decision-making standpoint, it would seem to matter only when the drop suggests that the model's performance is inferior compared with the performance of another prediction method. Among these choices would be quoting an overall average for all patients (ie, using no real model at all). However, this approach has a concordance index of 0.5. Another alternative would be to assess the prognostic factors and assemble them into an estimate using clinical judgment. Because of numerous human biases, clinical judgment generally underperforms statistical-prediction rules.3 Consequently, validating a prediction model may not be terribly useful unless it is being compared with an alternative method. Otherwise, what is validation versus invalidation? How much of a decrease in the concordance index constitutes invalidation, and when can a model be considered validated, given that the next sample drawn may reveal a very low concordance index? Although comparing concordance indices of competing models facilitates decision making, understanding the change in the concordance index still holds academic interest. What causes a concordance index to drop when a nomogram is tested? The nomogram is an equation with weights for variables; the user plugs in a variable's value (eg, tumor size), and this value is multiplied by the variable's weight. Two things can go wrong here: First, the variable (including the outcome variable being predicted) may be measured differently in the validation dataset. For example, symptom status is a subjective variable in the nomogram and was assessed by a single surgeon when the Kattan nomogram was developed. The nomogram would not perform as well if this variable were measured differently in the validation dataset. A second problem may be that the weights differ in different populations. Although we often believe that this is the case (eg, a model needs to be validated in a specific population), why should we expect the weights necessarily to be different? Note that this is a different question from the case mix, which, if the relevant predictors are included in the nomogram, should not affect the predictive accuracy that much if patients with more advanced disease or less advanced disease are identified in the validation dataset. In other words, if the weights are correct, then higher variable values (eg, tumor size) in a validation dataset should not have a strong adverse effect on nomogram predictive accuracy; the nomogram should perform proper case-mix adjustment. We need more accurate prediction models for patients with renal cell carcinoma. This is easier said than done, but at least 4 strategies may achieve that objective. One strategy is better modeling of the data. When constructing the Kattan et al. nomogram, 8 different approaches were compared, and a Cox regression model fared the best. However, many modeling approaches exist, and another approach may have done better. A second strategy toward improved prediction is to assemble larger datasets. There is a decreasing marginal return from this, but it is hard to know where we are on the predictive-accuracy curve given a particular sample size. A third strategy is to collect data more systematically and cleanly (eg, tighten the definition of symptom status). Finally, simply identifying novel markers that add incremental predictive accuracy obviously helps but is difficult. Future research should pursue these strategies and hopefully will provide more accurate prognostic models for physicians and patients.
No takes yet. Share an insight, caveat, or question.
Michael W. Kattan (2006) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: