Efficacy of statistical and design interventions for improving reproducibility in biomedical research

Over the past two decades, failures to replicate published findings in biomedical research have led to a perceived reproducibility crisis. Several interventions at the level of inferential statistics and experimental design have been proposed for improving reproducibility. These include raising the power by increasing the sample size, redefining the threshold for statistical significance, focusing on a minimally important difference, adding a penalising term for between-study variation, and combining data from multiple study centres. We used simulations to investigate the effectiveness of these interventions on measures of reproducibility. Artificial data sets were created by repeatedly sampling from defined parametric distributions based on published biomedical studies (i.e. real-world data sets). Reproducibility measures included the probability of positive findings, the probability of consistent inferential outcomes and the chance of capturing the true effect size. The results were mixed: some of the proposed interventions could improve reproducibility, while others had detrimental effects on reproducibility. Notably, the interventions affected the various aspects of reproducibility differently.