Correcting for Multiple Testing During Diagnostic Accuracy Studies

During diagnostic accuracy studies, hypotheses are often tested on more than one accuracy measure. One common example is to make inferences both on the area under the receiver operating characteristic curve (AUC) and on the sensitivity and specificity at a particular operating point. This represents a unique multiple testing situation as the AUC is a function of the sensitivity and specificity. Through a large simulation study, we showed that a naïve approach of testing all three accuracy measures at a 0.05-level greatly elevates the family-wise Type I error rate (i.e., up to 10%). When a gatekeeping approach to controlling the FWER was appropriate (family 1: AUC, family 2: sensitivity and specificity), adjustment in the second family was necessary to control the FWER across all scenarios. When a nonhierarchical approach was appropriate, the methods of Xie (2012 Xie, C. (2012), “Weighted Multiple Testing Correction for Correlated Tests,” Statistics in Medicine, 31, 341–352.[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]) offered increased power over Holm's step-down procedure, but required assumptions about the correlation among the test statistics. In our simulations, the correlation between the test statistics of the AUC and sensitivity and between the test statistics of the AUC and specificity ranged from 0.3 to 0.6, while there was no correlation between the test statistics of the sensitivity and specificity.