Comparative Effectiveness Research It Is Not

To the Editor:I take issue with Memtsoudis and Liu’s editorial acceptance1 of the study by Mashour et al.2 being referred to as comparative effectiveness research (CER), or the actual study being labeled as a practical clinical trial. It is neither. Although the study did indeed enroll a large, “real-world” patient population undergoing various procedures, the hospital setting was specific, and even more importantly, the interventions being compared could hardly represent what the Institute of Medicine expects of alternative treatment methods used in typical daily practice to be evaluated with CER.3 Put simply, the study results are not generalizable.CER is the direct comparison of existing (that is, commonly used) interventions, aiming to determine which treatment works best for whom and under what circumstances.3 An important ingredient of CER is pragmatic (or practical), effectiveness trials that reflect everyday clinical practice.3–6 The explicit purpose of pragmatic trials is to inform decision makers about study interventions commonly used in practice and to measure clinically important outcomes in typical populations.4,7–11 The design and interpretation of large pragmatic trials has been a major interest of mine for more than 15 yr.9,10,12 Pragmatic trials test commonly used interventions in a wide variety of healthcare settings—that is, they should represent “real-world” practice.The study by Mashour et al.2 occurred in a single university hospital system (albeit with three hospitals) in one city in the United States which used an electronic perioperative information system in all of its operating rooms. Furthermore, they used automated real-time analysis of bispectral index (BIS) values or minimum alveolar concentration every 5 min and provider-specific electronic alphanumeric paging alerts sent in under 1 min. This does not represent real-world practice in most of the western world (United States included). The most widely used approaches anesthesiologists use to avoid awareness include delivery of an appropriate concentration of hypnotic and other drugs, typically titrated to autonomic signs focusing on blood pressure and heart rate, and to look for patient movement—this is real-world monitoring of anesthetic depth. It is hard to see how the results of the study by Mashour et al.2 can be applied outside of their specific setting. In contrast, the B-Unaware trial at least tested two broadly accessible, competing interventions.13Memtsoudis and Liu go on to describe the B-Aware trial,14 for which I was the principal investigator, as “a traditional randomized trial” despite it clearly being a pragmatic, effectiveness trial. Unlike the study by Mashour et al.,2 the B-Aware trial was conducted in a much broader range of hospital settings, involving 21 hospitals in 5 countries. The decision to use a high-risk population in the B-Aware trial should not be confused with the bias inherent in the strict selection criteria seen in some trials, which are intended to exclude high-risk patients. Enrolling an at-risk cohort does not infer a lack of generalizability, for it has been repeatedly shown that study results (relative effect) are nearly always in the same direction and of similar magnitude in unselected settings8,15; in fact, a high-risk study population may underestimate the treatment effect.16,17 The decision to use a high-risk group adds to study power because of the increased event rate and so it is more efficient; no more, no less. Finally, the view that we did not compare BIS with an alternative strategy to decrease the risk of intraoperative awareness is blatantly wrong. As stated above, the near-universal approach to avoiding intraoperative awareness—in other words, a relevant “competing intervention”3—is traditional monitoring of (predominantly) patient movement, hypnotic agent delivery, and autonomic signs. It is hardly a “no intervention” group. The B-Aware trial, therefore, was a pragmatic trial, and it remains highly relevant to inform CER.Finally, CER is not only the generation of relevant evidence but also the synthesis of that evidence, with the latter best achieved using systematic review.3,6 I therefore provide results of broadly applicable CER evaluating the evidence for BIS monitoring in anesthesia, using an updated systematic review and meta-analysis of the pertinent randomized trials using Revman 5.1 software (Cochrane IMS team; fig. 1). This analysis includes input of the most robust data from the study by Mashour et al.,2 based on their as-treated data, because the presumptions of an intention-to-treat analysis cannot be claimed because of the gross failure of the intended intervention in 36% of the BIS group.There are six pertinent trials2,13,14,18–20 and one large observational study.21 The results of this CER approach depend on the local setting or scenario being considered:These up-to-date findings can be used by all anesthesia providers to determine whether or not to rely only on traditional monitoring, avoid total intravenous anesthesia because of the inability to use end-tidal volatile agent monitoring, with or without activation of the volatile agent monitoring alarms, use BIS monitoring, or purchase/configure an electronic monitoring system with automated paging alerts. Scenario (a) would be most relevant to most anesthesia providers in the western world.Another important component of CER is to consider efficiency: is the problem clinically important and is the intervention worthwhile and cost-effective?3,6 Regard for patient values and expectations, and consideration of benefits and harms, acquisition costs, and the number needed to treat should assist in such clinical and policy decision-making.There have been dozens of high-quality large clinical trials guided by principles of CER in perioperative medicine over the past decade, including in the field of anesthesiology. The shame is that these are often overlooked or otherwise not incorporated in clinical practice more quickly. That is what our patients expect.