Statistical testing in treatment by replication designs: Three options reconsidered

Burgoon, Hall, and Pfau (1991) offer an empirical comparison of three methods for statistical analysis of treatment by replication experiments involving message variables. Message replications within such designs might be: (1) ignored in the statistical analysis; (2) treated as levels of a fixed effect; or (3) treated as levels of a random effect. Burgoon et al. argue that although the three methods do not differ noticeably in practice, the third method is prone to both alpha‐inflation and low power. This paper examines the mathematical relationships among the three methods, specifies the empirical conditions under which they should be expected to yield different conclusions, and reviews evidence pertinent to the status of these conditions within the social influence literature. Contrary to the conclusions of Burgoon et al., there is good reason to expect that the three methods will often produce different results. Although the third method yields low power under certain conditions, under other conditions it is the only method of the three that offers a reasonable test of the treatment main effect.