Most statistical inferences in psychology are based on frequentist statistics, which rely on sampling distributions: the long-run outcomes of multiple experiments, given a certain model. Yet, sampling distributions are poorly understood and rarely explicitly considered when making inferences. In this article, I demonstrate how to use simulations to illustrate sampling distributions to answer simple practical questions: for instance, if we could run thousands of experiments, what would the outcome look like? What do these simulations tell us about the results from a single experiment? Such simulations can be run a priori, given expected results, or a posteriori, using existing datasets. Both approaches can help make explicit the data generating process and the sources of variability; they also reveal the large variability in our experimental estimation and lead to the sobering realisation that, in most situations, we should not make a big deal out of results from a single experiment. Simulations can also help demonstrate how the selection of effect sizes conditional on some arbitrary cut-off (p≤0.05) leads to a literature crammed with false positives, a powerful illustration of the damage done in part by researchers’ over-confidence in their statistical tools. The article focuses on graphical descriptions and covers examples using correlation analyses, percent correct data and response latency data.
Guillaume A. Rousselet (Tue,) studied this question.