On the Scientific Evaluation of Diagnostic Procedures

Diagnosis is the cornerstone of modern clinical medicine. It requires accurate observation and rational deduction. In its most complete form it is the process of identifying a disease by consideration of the history, symptoms, physical signs, and results of every other type of examination of the patient; it includes differential diagnosis and provides a basis for prognosis. It is realized, however, that often the process contains some element of uncertainty so that diagnosis might be defined as “the procedure of reaching the most probable conclusion based on the facts at hand.” This element of uncertainty extends to all branches of medicine. In those branches in which it has been measured—for example in pediatrics, clinical laboratory work, and chest surveys—it is found to be significant in degree, and therefore worthy of attention. As a result of this attention, methods for its correction or reduction can be evolved. Roentgenologic examination of the chest is one of the most important diagnostic procedures developed in the last half century. The results of its use are highly reliable so long as they are integrated with the other clinical findings, the history, the symptoms, and the laboratory data. When the method is used as the sole examination, however, as occasionally happens in everyday practice, its reliability may be evanescent. In recent years, roentgen examination of the chest as employed in mass radiography has been called upon by some to serve as a decisive diagnostic test. This has put an undue stress on the procedure. Further, instead of the medium usually employed in clinical practice, a small film process has been generally utilized. In order to investigate the relative diagnostic efficiency of the various roentgenographic and photofluorographic technics used in mass-survey work, a Board of Roentgenology was appointed in 1944 (4). This Board was made up of three chest specialists and two radiologists, and had the services of a biostatistician. Each member made independent interpretations of four sets of films and found to his astonishment that not only did he differ from his colleagues in apparently simple interpretations, but that he even differed with himself in a significant percentage of the same films which he read on two separate occasions. The Board recommended that this phenomenon be sUbjected to extensive and detailed investigation. This paper constitu tes a summary of certain phases of the investigation thus far pursued, and of certain deductions arising from this work. Observations In Other Fields In all human activities the “human equation” is present to a greater or lesser degree. In nearly every activity which can be tested, it has been repeatedly demonstrated that humans, even experts in a given field, exhibit enormous variations in their ability to be consistent with themselves and others equally competent (7, 10, 11).