Statistical Tests and Surveys of Power: A Critique

Cohen (1962) and Brewer (1972) surveyed statistical tests reported in the Journal of Abnormal and Social Psychology and the American Educational Research Journal, respectively, to determine the amount of power of those tests for the various sample sizes used and various magnitudes of effect sizes. Both authors concluded that the levels of power were low, and each author made negative comments about the situation or state of the art. Brewer went much further, even suggesting that two-thirds of the researchers surveyed would do better by flipping coins and not spending the time in data collection. In addition, he interpreted power as indicating the probability of a valid rejection of the null hypothesis after the statistical test was performed. Cohen (1973) applauded Brewer's survey, but was critical of most of Brewer's interpretations and recommendations, mostly relating to power after the fact and Brewer's alternate reporting schemes. Dayton et al. (1973) also pointed out the conditional nature of power. In their discussion they refer to the Bayesian inference model and while not necessarily intending to imply that Brewer is correct if interpreted Bayesianly, some readers might infer this from their comments.