Randomization Inference and Bias of Standard Errors

A nonparametric statistics course typically includes material regarding “distribution free” randomization based inference. However, the accuracy of reported p values and confidence intervals often relies on an unverifiable assumption of unit(subject)-treatment additivity. This assumption is not always explicitly stated in texts and, when the assumption does not hold, the implications on inference are seldom discussed. The focus of this article is the bias of standard errors of estimated mean treatment effects in the presence of nonadditivity. This bias is characterized and interpreted for a usual estimator of standard error in three common experimental designs: a two-sample completely randomized design, a matched-pairs design, and a balanced, two-period cross-over design. Even in the presence of nonadditivity, useful conservative estimates of a mean treatment effect can be obtained. This is illustrated using some previously published data.