Some advantages of permutation tests.

AbstractThere are two especially useful models of statistical inference, although only one, the model, is universally taught. The less well known permutation model is contrasted with the model and the case made that the permutation model is often more appropriate for the analysis of psychological data. Inappropriate interpretations generated by teaching only the model are illustrated. It is recommended that both models be taught so that students and applied workers have a better chance both of understanding the nature of the hypothesis that is being tested, and of correctly discriminating the statistical conditions that support causal inference and generality inference.Nearly all statistical testing is introduced using the normal curve model that leads to analysis of data by t and F ratios and to a preoccupation with generalizability of research results. The model is so common that many applied workers assume that it is the only model for statistical testing. The model was originally designed for parameter estimation and expresses the null hypothesis in terms of population parameters. For example, the null hypothesis for assessing the difference between two means is often expressed as Equation. As we all know, the ideal conditions for using the model to test statistical hypotheses such as this involve cases randomly selected from normally distributed parent populations with equal variances.Even textbook authors who introduce statistical testing with the binomial test emphasize the necessity of random sampling from a specified population. In most empirical research however, the concept of population enters statistical analysis not because the experimenter has actually randomly sampled some population to which he or she wishes to generalize, but because the only way researchers have been taught to interpret the results of statistical tests is in terms of inferences about populations.Kempthorne (1979) proposed that attempts to place all inferential situations in the model are misguided. He suggests that trying to encompass all types of investigation under one framework has led to the pooling of different types of investigation that have strongly different logical natures. For example, surveys and comparative experiments have different methods of data collection and different inferential goals. A major concern of surveys is external validity and generality inference, whereas comparative experiments are more concerned with internal validity and causal inference. Thus, we may need different statistical models for different research contexts.The Permutation ModelAn alternative to the model is the permutation or randomization model initiated by Fisher (1935) and developed by Pitman (1937a, 1937b, 1938). The permutation model is because no formal assumptions are made about the population parameters of the reference distribution, i.e., the distribution to which an obtained result is compared to determine its probability when the null hypothesis is true. Typically the reference distribution is a sampling distribution for parametric tests and a permutation distribution for many tests.(f.1)For many applied workers, nonparametric has been equated with rank tests. That is, either the data are ranks, or scores are transformed to ranks before conducting a statistical test. It is important to point out that the familiar rank tests, such as the Wilcoxon Rank - Sum test or Mann - Whitney U test, are members of a family of tests called permutation tests or randomization tests. It is less well known that there are two sets of permutation tests, those based on ranks and those based on scores. The tests on ranks traditionally have been named after the persons who were important in their development (e.g., Mann - Whitney, Wilcoxon, Kruskal - Wallis, Friedman). Often, tests of scores are referred to just as permutation tests although some reference has been made to the Fisher - Pitman test (e. …