Measuring Performance in Chest Radiography

PURPOSE: To use a standardized set of chest radiographs to quantify interobserver differences and to provide a basis for comparing the diagnostic performance of physicians. MATERIALS AND METHODS: A standardized set of 60 chest radiographs was presented to 162 study participants. Each participant reviewed the radiographs and recorded his or her diagnostic impression by using a fixed five-point scale. These response data were used to generate receiver operating characteristic curves and to establish performance benchmarks. The variations in performance were tested for statistical significance. RESULTS: Significant interobserver variability was identified during these assessments. The composite group of board-certified radiologists demonstrated performance superior to that of the radiology residents and nonradiologist physicians. CONCLUSION: By using a receiver operating characteristic approach and a standardized set of chest radiographs, observer accuracy and variability are easily quantified. This approach provides a basis for comparing the diagnostic performance of physicians. When value is measured as a diminution in uncertainty, board-certified radiologists contribute substantial value to the diagnostic imaging system.