A Redemption Song for Statistical Significance

Disagreement is a common occurrence in the field of statistics. In the last century, Ronald Fisher focused on the data-generating probability model known as the null hypothesis. Jerzy Neyman and Egon Pearson elaborated on Fisher’s null model, incorporating alternative data-generating probability models. Bayesians, such as Harold Jeffreys, mathematically combined subjective probabilities with the objective ones derived from the data. In the current century, these classical methodologies have been surpassed by modern, computer-intensive machine learning algorithms utilizing massive datasets, requiring implementation with advanced calculus and interpretation informed by domain-specific knowledge. This paper does not attempt to unify statistical theories, predict the future of statistical science, claim superiority for any methodology, or advocate for a radical methodological paradigm shift to qualitative research. This paper focuses on Fisher's statistical significance and the null hypothesis model. Computer-simulated data sets with different sample sizes were used to test a true null hypothesis of zero difference between two independent population means with independent samples t-tests. Statistical significance was determined with a 5% cut score for p-values, and substantive significance was evaluated with Cohen’s “effect size index d.” The results demonstrate that statistical significance is a viable tool for filtering out false effect sizes (effect size errors) that would otherwise be misinterpreted as substantively significant.