Systematic Reviews and Meta-Analyses in Regional Anesthesia and Pain Medicine

See article pages 379 and 395 In 1997, John C. Bailar III published an editorial in the New England Journal of Medicine entitled “The Promise and Problems of Meta-Analysis.”1 In that paper, Bailar acknowledged that meta-analysis held great potential for revolutionizing the field of medicine, but he also cautioned that meta-analyses were likely to be performed poorly, which would have serious implications. In particular, he was concerned that researchers would focus on the pooled mean effect size for a set of studies and ignore the fact that the effect of a treatment might vary across populations. At the time, I thought Bailar was being overly pessimistic. However, over the 25 years since his editorial was published, I have come to realize that he was correct to be concerned. The majority of published meta-analyses do make the mistakes he warned about. Indeed, some of these mistakes have even been codified and recommended in guidelines. For that reason, I am pleased to see the two papers2,3 in this issue of Anesthesia some might assume that the effects fall as much as 30 mg from the mean; and some might assume that the effects fall within 10 mg of the mean. The clinicians would be trying to reach a consensus, but each of them would be working with a different understanding of the facts. And it is not their fault. Given that I2 is 97%, any of these distributions is possible. So, how widely does the effect size vary? As explained in Part 2, the statistic that does provide this information is the prediction interval, which (if the analysis includes a sufficient number of studies) may be estimated as the mean effect +/- 2 standard deviations.7 In this analysis the prediction interval is estimated as −43 to +7 mg. This tells us that in 95% of cases, the true impact is expected to fall between 43 mg in favor of ESPB at one extreme, and 7 mg in favor of control at the other. Thus, there are relatively few cases where the effect meets the criterion for being clinically useful (30 mg), and a small number where it might be harmful. Clinicians might still have different opinions about whether this treatment should be pursued. That discussion is necessary and welcome. However, when we are working with the prediction interval, the discussion would be based on the actual results.8,9 MYTH # 3: WE SHOULD CLASSIFY HETEROGENEITY AS BEING LOW, MODERATE, OR HIGH BASED ON I2 There is a common practice of using the I2 statistic to classify heterogeneity as being low, moderate, or high based on cutoffs such as 25%, 50%, and 75%. This practice should be abandoned. As discussed above, the I2 statistic does not tell us how much the effect size varies. Therefore, any classifications based on I2 cannot tell us how much the effect size varies. In the ESPB example where I2 was 97%, this approach would classify the heterogeneity as “high,” when in fact the clinical impact of the treatment was mostly consistent across studies. By contrast, the prediction interval does tell us how much the effect size varies. Critically, it reports this on the same scale as the effect size itself—in units that are clinically meaningful. The statement: “In some 95% of cases the true impact is expected to fall between 43 mg in favor of ESPB (at one extreme) and 7 mg in favor of control (at the other)” tells us what we need to know in language that is clear and concise.10 The prediction interval is the statistic that addresses the question we have in mind when we ask about heterogeneity, and that researchers often (incorrectly) believe is addressed by I2. A FRAMEWORK FOR READING THE TWO HIGHLIGHTED PAPERS Once we abandon these myths, we can recommend a framework for thinking about the two papers highlighted herein. In some settings, our goal will be to assess the impact of a specific intervention, in a specific population, under a specific set of circumstances. In that case, we would define the kinds of studies very narrowly as discussed in the section on PICO (population, intervention, control, and outcomes) in Part 1. We would expect that any variation in the observed effects would be due primarily to random sampling error, and that the true effect size, from a clinical perspective, is essentially the same in all studies. As discussed in Part 2, the way we define “essentially the same” is not based on the I2 statistic. Rather, it is based on a prediction interval. If the effect size does turn out to be essentially the same in all studies, we would be able to report this (common) effect size with good precision. However, the results would only apply to this population and this variant of the intervention. In other settings, our goal will be to assess the impact of an intervention across populations, across variations of the intervention (eg, dose, mode of administration), across circumstances (type of surgery), and so on, as discussed in Part 1. In this setting, we will likely expect the effect size to vary across studies and our goal would be to describe the distribution of effects. Again, as discussed in Part 2, the way to do this is to report the prediction interval and then consider the clinical implications. When there is clinically important heterogeneity, the final step would be to identify moderators, such as patient type or dose, which are associated with that heterogeneity. If we have coded the data for these moderators, it may be possible to identify those that are related to the effect size. However, as explained in Part 2, these relationships or associations are observational and not causal. In summary, the 2 papers provide an excellent resource for planning and performing a systematic review and meta-analysis. However, to fully understand these papers, readers need to approach them with an open mind, and to recognize that much of what they have been taught about meta-analysis may be incorrect. In this editorial, I elected to focus on myths related to heterogeneity, but there are also myths related to publication bias, statistical models, subgroup analyses, and other aspects of meta-analysis. I address those in a PDF that can be downloaded for free at https://www.Meta-Analysis.com/anesthesia.4 DISCLOSURES Name: Michael Borenstein, PhD. Contribution: This author conceived and wrote the editorial. This manuscript was handled by: Thomas R. Vetter, MD, MPH, MFA.