Controversies Around the Role of Statistical Tests in Experimental Research

In spite of the widespread use of significance testing in empirical research, its interpretation and researchers ' excessive confidence in its results have been criticized for years. In this paper, we first describe the logic of statistical testing in the Fisher and Neyman-Pearson approaches, review some common misinterpretations of basic concepts behind statistical tests, and analyse the philosophical and psychological issues that can contribute to these misinterpretations. We then revisit some frequent criticisms against statistical tests and conclude that most of them refer not to the tests themselves, but to the misuse of tests on the part of researchers. We agree with Levin (1998a) that statistical tests should be transformed into a more intelligent process that helps researchers in their work, and finally suggest possible ways in which statistical education might contribute to the better understanding and application of statistical inference. Empirical sciences, in general, and particularly psychology and education, rely heavily on establishing the existence of effects using the statistical analysis of data. Statistical inference dates back almost 300 years, but statistical tests were systematized through the works of Fisher, Neyman, and Pearson and today most researchers implicitly use a mixture of the logics suggested by these authors. However, since the logic of statistical inference is difficult to grasp, its use and interpretation are not always adequate and have been criticized for nearly 50 years. For example, Yates (1951) suggested that scientists were paying too much attention to the tests ’ results and were forgetting the estimates of the magnitudes of the effects they were investigating. An early extensive review of criticisms against significance testing can be found in Morrison and Henkel (1970). More recently, a comprehensive summary of these debates, and alternative approaches suggested, has been provided by Harlow, Mulaik, and Steiger (1997).