Over the last twenty years, education researchers have increasingly conducted randomized experiments. Such experiments are usually designed in the hope of supporting comparisons of cost effectiveness across interventions, for example by using broad, standardized outcome measures. However, small effect sizes, wide confidence intervals, and heterogeneous treatment effects have limited their practical usefulness. We argue that, rather than directly informing decisions, randomized experiments should primarily play the role of testing theoretical models, which can in turn inform educators’ mental models. Since comparable effect sizes are not essential when testing theory, researchers can improve the power of such studies in various ways, including through the use of outcome measures more closely aligned with the theoretical constructs of interest. We also suggest that quasi-experimental methods may often be more suitable for informing education decisions, on the grounds that they can achieve greater precision at lower cost, with relatively moderate risks to internal validity.
Sims et al. (Fri,) studied this question.