A randomization-based perspective on analysis of variance: a test statistic robust to treatment effect heterogeneity

Fisher randomization tests for Neyman’s null hypothesis of no average treatment effect are considered in a finite-population setting associated with completely randomized experiments involving more than two treatments. The consequences of using the |F| statistic to conduct such a test are examined, and we argue that under treatment effect heterogeneity, use of the |F| statistic in the Fisher randomization test can severely inflate the Type I error under Neyman’s null hypothesis. We propose to use an alternative test statistic, derive its asymptotic distributions under Fisher’s and Neyman’s null hypotheses, and demonstrate its advantages through simulations.