Statistical and Substantive Significance of Pearson Bivariate Correlation Coefficients Under True or False Null Hypotheses

Modern, massive digital data requires computer-intensive algorithms (data science) for analysis. However, small data sets continue to be analyzed with classical, inferential statistical methods. Regrettably, these methods have been tainted by the abuse, misuse, and misunderstanding of statistical significance. Understanding statistical significance requires an appreciation of theoretical sampling distributions of summary statistics under a true null hypothesis. This paper demonstrates a method to teach statistical and substantive significance with empirical computer-simulated sampling distributions of Pearson’s correlation coefficients. Sampling distributions of Pearson’s correlation coefficients and p-values reveal that statistical significance with small sample sizes filters out effect size errors that would otherwise be considered substantively significant under a true null hypothesis.