It is argued that it is preferable to evaluate probabilistic diagnosis systems in terms of utility (patient benefit) or loss (negative benefit). We have adopted the provisional strategy of scoring performance as if the system were the actual decision-maker (not just an aid to him) and argue that a rational figure of merit is given by the average loss which patients would incur by having the system decide on treatment, the treatment being selected according to the minimum expected loss principle of decision theory. A similar approach is taken to the problem of evaluating probabilistic prognoses, but the fundamental differences between treatment selection skill and prognostic skill and their implications for the assessment of such skills are stressed. The necessary elements of decision theory are explained by means of simple examples mainly taken from the acute abdomen, and the proposed evaluation tools are applied to Acute Abdominal Pain data analysed in our previous papers by other (not decision-theoretic) means. The main difficulty of the decision theory approach, viz. that of obtaining good medical utility values upon which the analysis can be based, receives due attention, and the evaluation approach is extended to cover more realistic situations in which utility or loss values vary from patient to patient.
Habbema et al. (Wed,) studied this question.