The Covariance Decomposition of the Probability Score and Its Use in Evaluating Prognostic Estimates

The probability score (PS) or Brier score has been used in a large number of studies in which physician judgment performance was assessed. However, the covariance decomposition of the PS has not previously been used to evaluate medical judgment. The authors introduce the technique and demonstrate it by analyzing prognostic estimates of three groups: physicians, their patients, and the patients' decision-making surrogates. The major components of the covariance decomposition--bias, slope, and scatter--are displayed in covariance graphs for each of the three groups. The decomposition reveals that whereas the physicians have the best overall estimation performance, their bias and their scatter are not always superior to those of the other two groups. This is primarily due to two factors. First, the physicians' prognostic estimates are pessimistic. Second, the patients place the large majority of their estimates in the most optimistic category, thereby achieving low scatter. The authors suggest that the calculational simplicity of this decomposition, its informativeness, and the intuitive nature of its components make it a useful tool with which to analyze medical judgment.