Measuring diagnostic accuracy in the absence of a "gold standard"

OBJECTIVE: As the nation debates issues of national health care reform, psychiatrists seek equal status with other medical colleagues. To defend psychiatry in the national arena, the accuracy of psychiatric diagnoses must be measured. Indexes of accuracy such as sensitivity and specificity provide valuable information, yet they are rarely computed because there is no "gold standard" with which to compare them. The goal of this article is to show how this problem can be overcome and to encourage nosologists to use accuracy statistics in assessing the adequacy of psychiatric diagnoses. METHOD: The authors reviewed the literature on medical decision making to find methodological approaches to assessing diagnostic accuracy in the absence of gold standards. RESULTS: A lack of such standards is not unique to psychiatry and has been addressed with a variety of novel analytic procedures. Although these methods differ in many respects, each recognizes that the conventional 2 x 2 table of interrater agreement does not provide enough data for estimating diagnostic accuracy. After defining the data needed, each method provides a mathematical model that estimates accuracy statistics and the prevalence of a disorder. Most of these methods are variants of latent class analysis. The authors reanalyzed data from one of the reviewed papers to show that similar inferences about accuracy of diagnoses could be drawn from a conventional latent class analysis. CONCLUSIONS: There are potential pitfalls in using latent structure methods, but their cautious use would provide valuable information for psychiatric nosology. These methods supplement, but do not replace, data about outcome, family history, laboratory studies, and other validating criteria in making accurate diagnoses.