The Fisher‐Pitman permutation test: A non‐robust alternative to the normal theory F test when variances are heterogeneous

The Fisher–Pitman permutation test is an increasingly popular alternative to the ANOVA F test. As a test of equality of distributions, the permutation test is very attractive because it retains its stated test size without any distributional requirements. As a test of equality of location parameters, however, the permutation test retains its stated test size only under equality of all nuisance parameters. This homogeneity requirement is well known, but often overlooked. As a result, the permutation test is sometimes recommended when, in fact, the distributional requirements are not satisfied. This article examines the robustness of the permutation test to violation of the homogeneity requirement. In particular, the size of the Fisher–Pitman permutation test of equality of means is compared to the size of the normal theory F test in small samples when variances are unequal. Normally distributed populations having equal means are assumed. The size of the permutation test is found to be smaller than the size of the F test when the ratio of the harmonic to the arithmetic mean of the sample sizes is small and vice versa when the ratio is large (i.e. near 1). In either case, the difference between the sizes of the two tests is relatively small, except for extreme heterogeneity. The result is based on a comparison of the moments of the permutation and normal theory sampling distributions and is supported by the results of simulation studies.