Sensitivity of Fisher's exact test to minor perturbations in 2 × 2 contingency tables

The two tailed Fisher's exact P value is extremely sensitive to small perturbations in 2 x 2 contingency tables. An example indicates that a 1 per cent increase in the denominator of one treatment group results in a 32 per cent drop in the exact P value, but a mere 0.1 per cent decrease in the treatment success rate. This is equivalent to the increase in significance obtained by a 20 per cent increase in the sample size of both treatments without changing the observed success rates. This drop results from small changes in the probabilities of unobserved events. A systematic evaluation of 920 pairs of similar contingency tables shows that these fluctuations occur frequently over a wide range of sample sizes and significance levels. Doubling the one tailed exact P value provides a more consistent measure of inferential strength. We discuss various chi-squared continuity corrections.