The Incompleteness of Probability Models and the Resultant Implications for Theories of Statistical Inference

It is argued that statistical testing has been overvalued because it is perceived as an optimal, objective, algorithmic method of testing hypotheses. Researchers need to be made aware of the subjective nature of statistical inference in order not to make too much of it. Examples are given of aspects of data that are ignored in the computation of p values but are relevant to their interpretation. These include the fit of the chance distribution, the presence of influential points, the possibilities for post hoc selectivity, the presence of expected and unexpected trends in the data, and the amount of sampling variability that is present. Researchers should be taught that although probabilistic reasoning is a deductive process, making inferences from data is not. There is always potentially relevant information available over and above that which has been taken account of by any p value. Indeed, it is noted that a probability can never characterize all the uncertainty regarding an event because of problems to do with self-reference. Theories of statistical inference need to be weakened to take these ideas into account. Statistical tests assess the implausibility of supposing that the sign of a difference can be explained away as due to sampling error and they work only to the extent that this implausibility can be inferred from the size of the test statistic. The process of inferring plausibility is a subjective one and is inconsistent with the Neyman-Pearson Theory of statistical testing and Fisher's notion of fiducial statistical tests. Probabilities need to be assessed in the context of what they have and have not taken into account and this weakens Bayesian statistical inferences as well as classical ones.