When alternative analyses of the same data come to different conclusions: A tutorial using DeclareDesign with a worked real-world example

Recent studies in psychology have documented how analytic flexibility can result in different results from the same dataset. Here we demonstrate a package in the R programming language, DeclareDesign, which uses simulated data to diagnose the ways in which different analytic designs can give different outcomes. To illustrate features of the package, we contrast two analyses of a randomised controlled trial (RCT) of GraphoGame, an intervention to help children learn to read. The initial analysis found no evidence that the intervention was effective, but a subsequent reanalysis concluded that GraphoGame significantly improved children’s reading. With DeclareDesign we can simulate data where the truth is known, and thus can identify which analysis is optimal for estimating the intervention effect, using “diagnosands”, including bias, precision, and power. The simulations showed that the original analysis accurately estimated intervention effects, whereas selection of a subset of data in the reanalysis introduced substantial bias, overestimating the effect sizes. This problem was exacerbated by inclusion of multiple outcome measures in the reanalysis. Much has been written about the dangers of performing reanalyses of data from RCTs that violate the randomisation of participants to conditions; simulated data make this message clear and quantify the extent to which such practices introduce bias. The simulations confirm the original conclusion that the intervention has no benefit over “business as usual”. In this tutorial we demonstrate several features of DeclareDesign, which can simulate observational and well as experimental research designs, allowing us to make principled decisions about which analysis to prefer.