A constrained formulation for the receiver operating characteristic (ROC) curve based on probability summation

We propose a principled formulation of the ROC curve that is constrained in a realistic way by the mechanism of probability summation. The constrained and conventional ROC formulations were fitted to 150 separate sets of rating data taken from previous observer studies of 250 or 529 chest radiographs. A total of 20 different readers had used either discrete or continuous rating scales to evaluate those chest cases for likelihood of separate specified abnormalities: interstitial disease, pulmonary nodule, pneumothorax, alveolar infiltrate, or rib fracture. Both ROC formulations were fitted separately to every set of rating data using maximum-likelihood statistical procedures that specified each ROC curve by normally distributed latent variables with two scaling parameters, and estimated the area below the ROC curve (Az) with its standard error. The conventional and constrained binormal formulations usually fitted ROC curves that were nearly indistinguishable in form and in Az. But when fitted to asymmetric rating data that contained few false-positive cases, the conventional ROC curves often rose steeply, then flattened and extrapolated into an unrealistic upward "hook" at the higher false-positive rates. For those sets of rating data, the constrained ROC curves (without hooks) estimated larger values for Az with smaller standard errors. The constrained ROC formulation describes observers' ratings of cases at least as well as the conventional ROC, and always guarantees a realistic fitted curve for observer performance. Its estimated parameters are easy to interpret, and may also be used to predict observer accuracy in localizing the image abnormalities.