Intersections of Statistical and Substantive Significance Under a True and False Null Hypothesis

This paper has two objectives: (1) Provide intuitive insight into statistical and substantive significance intersections with histograms, bar graphs, and crosstabs with data from independent samples t-tests. (2) Convincingly demonstrate with graphs and a few numbers that statistically significant p-values from independent samples t-tests are valuable for screening out standardized mean differences, known as Cohen’s d (effect size), that would otherwise be misinterpreted as substantively significant. The author hopes the empirical sampling distributions in this paper help students, applied researchers, and science writers to properly understand and appreciate the value of statistical significance for scientific inference and decision-making with small sample sizes (n < 1,000) in the face of uncertainty.