Comparison of visual grading and free-response ROC analyses for assessment of image-processing algorithms in digital mammography

OBJECTIVE: To compare two methods for assessment of image-processing algorithms in digital mammography: free-response receiver operating characteristic (FROC) for the specific task of microcalcification detection and visual grading analysis (VGA). METHODS: The FROC study was conducted prior to the VGA study reported here. 200 raw data files of low breast density (Breast Imaging-Reporting and Data System I-II) mammograms (Novation DR, Siemens, Germany)-100 of which abnormal-were processed by four image-processing algorithms: Raffaello (IMS, Bologna, Italy), Sigmoid (Sectra, Linköping, Sweden), and OpView v. 2 and v. 1 (Siemens, Erlangen, Germany). Four radiologists assessed the mammograms for the detection of microcalcifications. 8 months after the FROC study, a subset (200) of the 800 images was reinterpreted by the same radiologists, using the VGA methodology in a side-by-side approach. The VGA grading was based on noise, saturation, contrast, sharpness and confidence with the image in terms of normal structures. Ordinal logistic regression was applied; OpView v. 1 was the reference processing algorithm. RESULTS: In the FROC study all algorithms performed better than OpView v. 1. From the current VGA study and for confidence with the image, Sigmoid and Raffaello were significantly worse (p<0.001) than OpView v. 1; OpView v. 2 was significantly better (p=0.01). For the image quality criteria, results were mixed; Raffaello and Sigmoid for example were better than OpView v. 1 for sharpness and contrast (although not always significantly). CONCLUSION: VGA and FROC discordant results should be attributed to the different clinical task addressed. ADVANCES IN KNOWLEDGE: The method to use for image-processing assessment depends on the clinical task tested.