When possible, report a Fisher-exact P value and display its underlying null randomization distribution

Significance Statistical analyses of randomized experiments often rely on asymptotic P values instead of using the actual randomization procedure that led to the observed data. Fisher-exact and asymptotic P values can differ dramatically: The former should be preferred because it is calculated using the exact null randomization distribution, which, in small samples, can substantially differ from its approximating Student’s t distribution. Moreover, we may learn something scientifically interesting from examining the shape of the null randomization distribution.