Non-convergence in medical meta analysis: evidence suggestive of a replication crisis

The replication crisis in medicine has cautioned us not to base our medical inferences on a single study. Instead, replication studies have been widely argued to be a cure for many of the ills causing the replication crisis. The plausible thought is that later replication studies can contradict an initial result and meta analyses aggregating the available studies allow us to find – or at least get close to – the true effect size. Stegenga argued that this may not happen in practice. Here, I strengthen Stegenga’s claim by showing that not only may this fail to happen, but I also present evidence strongly suggesting that replication failures of meta analyses of intended treatment effects undermine their status as a reliable basis for medical decision making. I thus suggest that meta analysis may now also be facing a replication crisis. Finally, I discuss proposed remedies, arguing that while practical measures may mitigate some issues, medical research and the environment in which it is situated place inherent constraints on the reliability of meta-analytic inference which complicates medical decision making.