The winner's curse: why large effect sizes in discovery trials always get smaller and often disappear completely

In 2000, a randomised trial by Ronco et al. entitled Effects of different doses in continuous veno-venous haemofiltration on outcomes of acute renal failure was published [1]. Behind the nondescript title lay some dramatic results. Four hundred and twenty-five patients admitted to the intensive care unit (ICU) with acute renal failure were allocated at random to receive renal replacement therapy with continuous haemofiltration at one of three doses. The primary outcome was survival at 15 days after discontinuation of treatment. Remarkably, survival in the groups who received doses of 35 or 45 ml.kg-1.h-1 was much higher (57% and 58%, respectively) than in patients treated with 20 ml.kg-1.h-1 (41%), differences that were associated with very small p values. The authors concluded that, “…ultrafiltration should be prescribed according to patient's bodyweight and should reach at least 35 ml.kg-1.h-1.” In our ICU, we did exactly that and increased our standard dose of haemofiltration to 35 ml.kg-1.h-1 and for patients who were very unwell we used a dose of 45 ml.kg-1.h-1. Over the next few years, two smaller randomised trials failed to confirm the dramatic results observed by Ronco et al. [2, 3]. Then, in 2009, the results of the Randomised Evaluation of Normal versus Augmented Level (RENAL) Replacement Therapy trial were published [4]. In this trial, 1464 patients were allocated randomly to receive either higher-intensity (40 ml.kg-1.h-1) or lower-intensity (25 ml.kg-1.h-1) renal replacement therapy. The authors found no difference in 90-day mortality, which was 44.7% in each group. Chastened, we again rewrote our haemofiltration protocol. What happened? How could absolute differences of 16% and 17% observed in the trial by Ronco et al. [1] evaporate entirely in the RENAL Replacement Therapy trial [4]? Were the participants in the two trials very different? Was the original trial biased? Did the different mortality outcomes play a part? While it is possible that one or more of these factors were at play, there is no requirement to invoke differences in trial design or bias to explain the dramatically different results observed in the two trials. Rather, effect size shrinkage of novel discoveries in subsequent larger trials is an expected consequence of null hypothesis significance testing and arises from the fact that to cross the significance threshold (p ≤ 0.05) a small trial requires a bigger effect size than a larger trial [5]. Consider a patient population with a mortality risk of around 50%. Patients with severe acute respiratory distress syndrome might fit the bill. Now consider an intervention that has a real effect on reducing mortality by 10%. We can simulate a randomised trial evaluating such an intervention by drawing random samples from two binomial distributions, one with a success probability of 0.5 and another with a success probability of 0.4, and then doing a chi-square test on the sample pair (the statistical term ‘success probability’ is unfortunate when the outcome of interest is death). A random sample of 390 from a binomial distribution with a success probability of 0.5 tested against a random sample of 390 from a binomial distribution with a success probability of 0.4 has a power of around 80% to detect the real treatment effect at the 5% significance threshold, where ‘detect a real treatment effect’ means observing p ≤ 0.05. Since power is 80%, if we repeated the simulation 1000 times we would expect to observe p ≤ 0.05 about 800 times. If we take just the significant trials and calculate the absolute difference in each case we obtain a fairly good picture of the effect sizes we might expect from such a trial. We can also vary the sample size to achieve different levels of statistical power. The results of such an analysis are shown in Fig. 1 and online Supporting Information Table S1. We see from Fig. 1 that when the sample size is less than about 180 (90 per group, power 25%), the median absolute difference for significant trials is at least twice the true absolute difference. As the sample size (and power) increases, the effect size decreases, converging on the true value at a sample size of around 1000 (500 per group, power 90%). In the above example, the true population effect size is known. In real life, we have no way of knowing what the true effect size is or, indeed, if there is one. So, what happens when we simulate randomised trials from populations where there is no real treatment effect? Figure 2 and online Supporting Information Table S2 show the effect sizes for significant results obtained from simulated statistical testing from identical binomial distributions, each with a success probability of 0.5. The concept of statistical power does not apply for the null simulations, and we used the same sample sizes as for the simulations with a real effect. We see from Fig. 2 that when the sample size is less than about 180 (90 per group), significant results are associated with a median absolute difference of around 20%. As the sample size increased, significant results are associated with smaller effect sizes. When power is low or when there is no real treatment effect, few trials report significant results. For instance, when power is 20% we expect roughly 20% of trials to report p ≤ 0.05 at the 5% significance threshold. When there is no treatment effect, we expect roughly 5% of trials to observe p ≤ 0.05. Nevertheless, publication bias means trials reporting significant results skew the literature, suggesting the weight of published evidence favours a real treatment effect [6]. Publication bias is the failure to publish studies on the basis of the strength or direction of the results. While there is little evidence that manuscript acceptance rates are influenced by the findings, authors may choose not to seek publication of small or poorly designed trials reporting non-significant results [6]. Mandatory trial registration helps prevent p-hacking and other unethical behaviour but does not guarantee trials are published. By contrast, discovery trials—trials testing novel interventions and reporting statistically significant differences—are likely to be published and are often cited highly. Publication bias is a problem across medical research, including in anaesthesia [7, 8]. In a study of abstracts presented at the American Society of Anesthesiologists annual meeting, 58% of studies reporting positive results were subsequently published compared with 41% for negative studies (relative risk 1.42, 95%CI 1.22–1.66; p ≤ 0.001) [7]. In contrast to small studies, large, multicentre trials costing millions of pounds tend to be published irrespective of the findings. The trial by Ronco et al. is unusual in that the authors observed a significant result for a primary mortality outcome [1]. Most anaesthesia or ICU trials where mortality is the primary outcome report non-significant differences [9, 10]. However, mortality is often reported as a secondary outcome, and this can lead to misleading findings. For instance, in 2009 De Hert et al. reported a randomised trial comparing the effects of volatile anaesthesia (sevoflurane or desflurane) and total intravenous anaesthesia (TIVA) in patients undergoing coronary artery bypass graft surgery [11]. The trial involved 414 participants and the primary outcome was postoperative troponin release. While there was no difference in the primary outcome, one-year mortality was significantly different, being 12.3% in the TIVA group, 3.3% in the sevoflurane group and 6.7% in the desflurane group (p = 0.034). While probably a chance finding, the mortality difference was used as part of the justification for a multicentre trial involving 5400 patients. In the larger trial, mortality was 2.8% in patients allocated to volatile anaesthesia compared with 3.0% in those allocated to TIVA (p = 0.71) [12]. In the Vitamin C Infusion for Treatment In Sepsis Associated Acute Lung Injury (CITRIS-ALI) trial, Fowler et al. compared high-dose vitamin C to placebo in 167 patients with sepsis or acute respiratory distress syndrome [13]. While there was no difference in the primary outcome (the sequential organ failure assessment score) between the groups, differences in three of 46 secondary outcomes were statistically significant, one of which was 28-day mortality. Mortality was 46.3% in patients allocated to placebo compared with 29.8% in those allocated to receive vitamin C (absolute difference 16.5%, p = 0.03). The secondary outcomes in the CITRIS-ALI trial were exploratory and p values were reported without correction for multiple testing. Observing three of 46 (7%) significant outcomes due to random chance alone would not be unexpected at the 5% significance threshold. Notwithstanding the fact that the observed mortality difference in the CITRIS-ALI trial almost certainly arose by chance, it was used as part of the justification for the larger Lessening Organ dysfunction with VITamin C (LOVIT) trial, comparing the effect of high-dose vitamin C with placebo in patients whose lungs were mechanically ventilated as a result of sepsis [14]. Interestingly, the LOVIT trial found a statistically significant difference for harm in patients treated with vitamin C [14]. As we discussed in another paper, the significant result in the LOVIT trial is also most likely not reflective of a real treatment effect of vitamin C in patients with sepsis [15]. How should we respond to effect sizes reported in small discovery trials? First, and most obviously, we should recognise that the observed effect size is likely unrepresentative of the population effect size and that the true population effect size is almost certainly smaller—if one exists at all. Inflated effect sizes in discovery trials are much more likely when statistical power is low, and low power is endemic across medical research [16]. We can gain insight into the likely presence of low power and inflated effect sizes by examining the 95%CI on the observed treatment effect. First, if the 95%CI is wide and either boundary is close to the null value, it indicates a high degree of imprecision about the true effect size. Second, we should recognise that inflated effects are an inevitable consequence of null hypothesis significance testing, and do not necessarily imply bias in the way the trial was conducted or malfeasance by the authors. Third, we should be especially sceptical of significant differences in secondary outcomes, particularly when these are numerous and there is no correction for multiple testing. Sample sizes are calculated based on the primary outcome, and trials are typically underpowered to detect differences in secondary outcomes, particularly for mortality. Fourth, as we discovered more than 20 years ago, we should not change our practice on the basis of a single randomised trial, even if the trial observes a large difference in a hard outcome such as mortality. Finally, researchers should avoid using effect sizes observed in small discovery trials as the basis of the sample size calculation for subsequent larger trials. Using the inflated effect size from a small trial means the large trial will be underpowered to detect the true population effect size, increasing the chance of a non-significant result [17]. Basing sample size calculations on a minimum effect size of interest avoids the problem of underpowered multicentre trials [18]. The phenomenon where the true effect of an intervention is modest or non-existent but the effect size observed in a small discovery trial is large, is called the ‘winner's curse’ [19]. The curse is that what appears to be a win (a statistically significant result) is actually a loss, as the result is inflated or even in the wrong direction and, therefore, highly misleading. Effect size shrinkage in subsequent trials occurs by two mechanisms. First, by regression to the mean, whereby extreme random events are likely to be followed by less extreme events. Consequently, a subsequent small trial is likely to observe a smaller (i.e. non-significant) difference. Furthermore, the initial trial may have been done in an ideal population group and undertaken by centres which are optimised for the best possible implementation of the intervention, potentially magnifying the treatment effect. Second, a large follow-up trial is more likely to observe an effect size that is closer to the true population effect size (Fig. 1). In multicentre trials, heterogeneity amongst trial participants and between different centres may act to reduce the observed treatment effect. Either way, over time, published effect sizes get smaller, a phenomenon that has been termed ‘regression to the truth’ [20]. Effect size shrinkage has long been recognised. In the salons of Paris in the late 18th century, pyramidal elm bark was a popular restorative, taken for maladies involving the nerves, chest and stomach. When asked by an anxious patient if she should take it, her physician, Michel-Philippe Bouvart replied, “Take it, Madame, and hurry up, while it still cures” [21]. When faced with an implausibly large treatment effect observed in a small discovery trial, we can do no better than remember the words of Carl Sagan: “extraordinary claims require extraordinary evidence” [22]. Data source: simulations were done in R (R core team, 2022, R Foundation for Statistical Computing, Vienna, Austria). p values were calculated using the prop. test function of the stats package. Power calculations were done using the power.prop.test function of the stats package. All tests were two-sided. The code used to perform simulations and create the figures is available at https://www.github.com/cjdbarlow/papers. No external funding and no competing interests declared. Open access publishing facilitated by The University of Auckland, as part of the Wiley - The University of Auckland agreement via the Council of Australian University Librarians. Table S1. Risk differences for significant trials by sample size in the presence of a real treatment effect. Table S2. Risk differences for significant trials by sample size when there is no real effect. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.