Meta-analysis, Medical Reversal, and Settled Science

“Medical reversal” describes the publication of clinical data that contradict previously published data and establishes a new “evidence-based” standard. Worrisome at best (and harmful at worst), reversal is surprisingly common, and exists in nearly all medical specialties. Recent high-profile examples include vertebroplasty for spinal fractures1 and hormone therapy in postmenopausal women.2 The failure of initial reports of tight glucose control3 and daily sedative interruption4 to withstand subsequent scrutiny suggests that the evidence base underlying perioperative critical care may have a similarly unstable foundation. Although the reasons for reversal are incompletely understood, potential mechanisms include fraud, error, inadequate statistical analysis, mechanistic plausibility,5 and cognitive biases. Underlying many of these possibilities is a tacit understanding that the later article is “true,” and the earlier, “reversed” findings must have been incorrect. But 3 other frames are possible: the earlier studies were correct (and the later studies were wrong), both earlier and later findings were correct (implying that the phenomenon being studied has changed over time), or neither finding is correct, and the truth lies elsewhere. In seeking an explanation for reversal, the choice of frame is not always easy because an adequately powered randomized prospective study design may not protect the finding against the subsequent reversal.6 In this issue of Anesthesia & Analgesia, Kalra et al7 publish the results of a meta-analysis of targeted temperature management (TTM) (cooling) after cardiac arrest. The authors identified 11 trials from 1966 to 2016 containing 4782 patients. Individual trials varied in rates of bystander cardiopulmonary resuscitation, cooling protocol, and method. After multivariate random-effects modeling, the authors concluded that TTM had no effect on mortality rates or cerebral performance status. Anesthesia & Analgesia readers who are familiar with TTM research may wonder why another meta-analysis of TTM is needed. The history of TTM fits a classic reversal scenario: multiple small clinical trials with unclear consensus, followed by high-profile randomized clinical trials establishing a clinical “truth.” In the case of TTM, 2 small but high-profile trials were published in the same 2002 issue of the New England Journal of Medicine,8,9 with an accompanying editorial10 citing as evidence of correctness that the 2 studies were performed on different continents, produced similar effect sizes, and that a plausible mechanism existed for the protective effect. By 2010, cooling to 32°C–34°C was enshrined in American Heart Association guidelines for cardiopulmonary resuscitation11 and strongly recommended by 5 professional critical care societies.12 Then came the reversal. A 2013 study of 950 patients (>4 times the combined number of 213 in the initial 2002 trials) randomized patients to 33°C vs 36°C after cardiac arrest, and found no difference in survival or neurological outcome.13 Also published in the New England Journal, this study was intensely discussed, garnering an Altmetric score of 470 by January 201414 and 664 as of August 22, 2017, placing it in the top 5% of all research outputs scored by Altmetric. Despite this apparent reversal, little has changed regarding the recommended use of temperature management after cardiac arrest in 2017. Both the 2016 Cochrane15 and the 2015 American Heart Association updates16 continue to recommend postarrest cooling. Critical care and emergency medicine specialists argue over cooling method, duration, timing, target, and applicability. With TTM, the net result of literature reversal is more, not less cognitive diversity. In principle, meta-analysis is an ideal solution. By combining the results of several studies, meta-analysis increases statistical power, facilitates understanding of variability between studies, and allows generalization of results to a wider population. The TTM landscape, dotted with multiple small studies and divergent results, is tailor made for meta-analysis, and multiple groups have obliged with both negative17,18 and positive results.16 However, meta-analysis also presumes that an absolute truth underlies all studies, but it is detected with varying accuracy and precision. What if that assumption is wrong, and the story of TTM really is one in which one (or more) of the dueling studies is incorrect? Does it then make sense to lump the results of correct and incorrect studies together? In such cases, meta-analyses may result in less, not more, truth. Even when all studies are correctly performed, repeated meta-analyses raise an issue akin to that encountered during interim analyses of a randomized controlled trial. Stopping early when data are few and the likelihood of an outlier effect is high risks a type 1 error, whereas addition of multiple negative trials with high heterogeneity risks never reaching clear significance. Aware of this challenge, meta-analysts have responded with trial sequential meta-analysis (TSA), a Bayesian modification of standard meta-analysis in which thresholds for significance or futility are adjusted based on available data. With TSA, thresholds for significance are raised when few data exist (and the likelihood of a false positive is high) and lowered when a wealth of data are available (and the likelihood of identifying a real effect with further studies is low). A 2017 treatise on TSA19 includes a sample TSA-based analysis of TTM explaining why early meta-analyses may have found a benefit, why later efforts (including Kalra et al7) are more inconclusive, and why future trials or meta-analyses of TTM are likely to be futile. Deciphering whether the TTM story represents reversal or fertile ground for meta-analysis may not be easy. Kalra et al7 found considerable variation among studies in age, time to return of spontaneous circulation, percentage of bystander cardiopulmonary resuscitation, a large 12–28-hour variation in cooling duration, and multiple cooling techniques, including helmet, ice packs, and intravenous administration of cold fluid.11 Which of these elements may affect TTM results (if at all) is unknown. In a 2016 analysis of Cochrane reports, outcomes graded as having a high quality of evidence had no better concordance among individual trials than did outcomes graded as having a low quality of evidence.20 Observations such as these are disquieting because they suggest that only time can reveal the true answer. So what should readers of Anesthesia & Analgesia make of repeated meta-analyses of TTM (or any other topic)? First, just as readers are critical of individual studies, so should they be appropriately thoughtful about meta-analysis results. Questions of applicability, generalizability, and actual benefit (as opposed to control group harm) are germane to both types of data. To the above, we would add the possibility that meta-analyses in fields in which reversal has occurred may be incorporating the results of not just underpowered, but also flawed research. Absent a need to clarify 1 or more of the above questions, repeated meta-analyses, especially for low-certainty data, may be an exercise similar to “p hacking,” in which analyses are repeatedly run to extract a statistically significant finding, rather than to answer a hypothesis.21 Alternatively, repeat meta-analyses may serve a useful role in highlighting overlooked gaps in the data and pointing out questions for future investigation. Regardless, the role for repeated meta-analyses remains unclear, and such studies may confuse as much as they clarify. DISCLOSURES Name: Mark E. Nunnally, MD, FCCM. Contribution: This author helped prepare the manuscript. Conflicts of Interest: None. Name: Avery Tung, MD, FCCM. Contribution: This author helped prepare the manuscript. Conflicts of Interest: A. Tung is an executive editor, Critical Care and Resuscitation, for Anesthesia & Analgesia. This manuscript was handled by: W. Scott Beattie, PhD, MD, FRCPC.