Statistical significance does not imply a real effect

The overall purpose of the ‘Statistical Points and Pitfalls’ series is to help readers and researchers alike increase awareness of how to use statistics and why/how we fall into inappropriate choices or interpretations. We hope to help readers understand common misconceptions and give clear guidance on how to avoid common pitfalls by offering simple tips to improve your reporting of quantitative research findings. Each entry discusses a commonly encountered inappropriate practice and alternatives from a pragmatic perspective with no mathematics involved. We encourage readers to share comments on or suggestions for this section on Twitter, using the hashtag: #mededstats. The statement that a statistically significant outcome in a small sample must reflect a real effect is misleading and may result in overconfidence in findings obtained in small-sample studies. We recommend caution in drawing inferences for educational practice from small samples, and provide some practical tips for reporting at the end of this entry. We must remember that rejecting a null hypothesis is always accompanied with a chance that the null hypothesis was actually true. In a study, we take a sample to estimate results for the whole population from which the sample is taken. Means, correlation coefficients, and other sample findings tend to be imprecise estimates of the corresponding population parameters of interest when sample sizes are small [1]. As recognized by most researchers, small samples tend to have very limited statistical power for detecting population differences (e.g., between groups) and relations (i.e., between variables) of interest. That is, even if a rather substantial difference or relation exists in the population from which we sample, small samples often fail to obtain a statistically significant difference or relation. Consequently, using small samples, we will often fail to reject the null hypothesis of no difference or no relation when that null hypothesis is not true. This testing error is called a Type II error [1], familiar to most researchers. So when a small sample size produces a significant difference, researchers erroneously conclude, ignoring the possibility of Type II error, that the difference must reflect a real effect. However, researchers appear to be less aware of the fact that of all statistically significant findings obtained, a larger portion results in Type I errors (i.e., rejecting a null hypothesis that is true) in the case of small samples when compared with samples of a larger size. It is important to understand this issue, as it challenges the aforementioned misconception that a statistically significant outcome in a small sample must reflect a real effect. Let us, therefore, elaborate on this issue in a bit more detail.