When one performs multiple hypothesis tests, the amount of Type I errors one expects to make increases with the number of tests. In certain scenarios, it is preferable to keep the probability of making one or more Type I errors, the Family-Wise Error Rate, small. Traditional methods like Bonferroni correctionor Holm’s method ensure control of the FWER through critical values that remain valid even under the worst-case distribution of p-values. Although valid, they are conservative since they do not incorporate information about the dependency between the p-values. By using resampling methods one can implicitly incorporate those structures by estimating the joint null distribution of the p-values. We show that under the assumption of Subset Pivotality the resampling methods strongly control FWER, and using simulations we show that they are on average slightly more powerful than traditional methods when testing coefficients of a regression model.
Jonsson Samuel (Thu,) studied this question.