Effect of Observer Instruction on ROC Study of Chest Images

Receiver-operating characteristic (ROC) analysis has been used in many medical imaging applications during the past decade. In order to ensure that reader-confidence ratings are analyzable (well distributed to meet convergence requirements of curve-fitting algorithms) and meaningful (limit extrapolation of the data), many investigators train readers specifically for this purpose. No experimental data are available concerning the possible effects of such training on the results of ROC studies. We performed a multi-observer, multi-disease study in which 300 chest images were rated by four radiologists before and after they were trained to provide well-distributed confidence ratings. The results indicate that for our data set, reader and disease-specific accuracy was not significantly affected by the training process for interstitial disease and pneumothoraces. However, the accuracy of two readers was significantly affected for the detection of nodules (P less than 0.05), and the overall accuracy of one reader was significantly affected for the classification of normal versus abnormal images (P less than 0.01). Thus, in spite of the difficulties associated with the performance of ROC studies in a free-reading environment, one should carefully consider the possible effects of any intervention on the results prior to conducting ROC studies.