The Effect of Simulation Order on Level Accuracy and Power of Monte Carlo Tests

SUMMARY We discuss the effect of simulation order on level accuracy and power of Monte Carlo tests, in a very general setting. Both parametric problems, with or without nuisance parameters, and nonparametric problems are treated by a single unifying argument. It is shown that if the level of a Monte Carlo test is known only nominally, not precisely, then the level error of a Monte Carlo test is an order of magnitude less than that of the corresponding asymptotic test. This result is available whenever the test statistic is asymptotically pivotal, even if the number of simulations is held fixed as the sample size n increases. It implies that Monte Carlo methods are a real alternative to asymptotic methods. We also show that, even if the number of simulations is held fixed, a Monte Carlo test is able to distinguish between the null hypothesis and alternative hypotheses distant n– 1/2 from the null.