Exact Properties of Some Exact Test Statistics for Comparing Two Binomial Proportions

Abstract Using exact unconditional distributions, we evaluated the size and power of four exact test procedures and three versions of the χ2 statistic for the two-sample binomial problem in small-to-moderate sample sizes. The exact unconditional test (Suissa and Shuster 1985) and Fisher's (1935) exact test are the only tests whose size can be guaranteed never to exceed the nominal size. Though the former is distinctly more powerful, it is also computationally difficult. We propose an alternative that approximates the exact unconditional test by computing the exact distribution of the χ2 statistic at a single point, the maximum likelihood estimate of the common success probability. This test is a modification of the test of Liddell (1978), which considered the exact distribution of the difference in the sample proportions. Our test is generally more powerful than either the exact unconditional or Liddell's test, and its true size rarely exceeds the nominal size. The uncorrected χ2 statistic is frequently anticonservative, but the magnitude of the excess in size is usually moderate. Though this point has been somewhat controversial for many years, we endorse the view that one should not use Fisher's exact test or Yates's continuity correction in the usual unconditional sampling setting.