That Was Then, This Is Now! The Value of Observing Change

For the strongest evidence of "proof" in medicine, we rightly seek evidence from randomized controlled trials (RCTs), expecting the process of randomization to result in comparison groups equitable in all respects save the variable in question [1]. Unfortunately, such studies are not always available, for there are too many questions to ask, and the costs of study can be excessive. RCTs are particularly problematic when the event to be studied is infrequent or when the process of care is changing so rapidly that the requisite number of subjects cannot be enrolled before the question becomes obsolete! It can also be challenged whether the RCT is ethically acceptable, because rigid protocols preclude individualization of therapy, creating a conflict of interest for the clinician-scientist [2]. This factor, as well as other issues of generalizability of the results [3], raises questions as to whether the published results of a given RCT apply to patient populations outside the study itself. There is much to learn from experience. Hospitals generate protocols and care maps that reflect the best guess as to what will lead to the best patient outcome. There is seldom absolute proof of the optimal course, although scrutiny of past experience usually defines the success of the current regimen. Change to the established process follows on the suggestion of a "better way," perhaps consequent to a technological advance or to a problem such as poor results or resource restrictions. The medical team adjusts, evaluates the result, and compares it with the previous data set. It is a scenario that is played out regularly throughout clinical practice, occurs at the local level, and stimulates the renewal of excellence embraced by terms such as "continuous quality improvement." In reality, the process is another form of observational research. When do data from an observational study achieve the standard necessary to become incorporated into one's evidence-based medical practice? Such studies have certainly met the standard in years past: John Snow did not randomly remove handles from water pumps, but rather compared cholera infection rates before and after taking action [4]. The National Halothane Study [5] is a classic example of epidemiology applied to anesthesia, and it exonerated a most useful drug in comparison to the historic alternatives. The sequential observations from New South Wales in Australia [6] illustrate the improvement in outcome associated with the better training of those providing anesthetic care. These few examples illustrate the usefulness of observational methodology in answering questions that could not be addressed in RCTs. In this issue of Anesthesia & Analgesia is an example of a observational study [7] in which patients undergoing abdominothoracic esophagectomy were managed with a multimodal approach to intra- and postoperative management, including personalized acute pain therapy. Their results were compared with those of another group of subjects undergoing the same procedure in the preceding 8-mo period, but with epidural therapy confined to the postoperative period without personal titration. The outcomes related to use of intensive care resources, ambulation, and return of function were improved by using the new approach, although the late medical complications were not significantly different from those of the historic controls. No economic data are given, but the authors speculate that such therapy can provide economic advantages as well. Can the results of such an experimental design be accepted? The interpretation of observational studies differs from that of properly randomized trials (Table 1). The greatest potential weakness of observational studies, particularly those using historic controls, rests with the comparability of the patient groups. In the present example, Brodner et al. have done well to describe the equivalence of the two groups with respect to smoking and alcohol consumption, as well as the presence of concurrent disease of other major organ systems. Unfortunately, authors can seldom identify all possible confounding influences; randomization is the only satisfactory method, but it is not possible with historic comparisons. Suspicion always lurks that some unidentified aspect actually caused the difference between groups, rather than the variable embraced by the specific hypothesis.Table 1: Characteristics of Observational and Controlled Research DesignsOther unique aspects of the site of study, best described as administrative factors, may preclude the reader from accepting observational results as applicable to one's practice. For example, the fast track that led to earlier extubation and ambulation may only be possible with sufficient levels of house staff coverage. As another example, anesthetic studies invariably are undertaken in patients undergoing surgical procedures, yet they seldom include surgeon expertise as a source of bias. Rates of medical morbidity are reduced by hospital characteristics [8] and by experience of the operators [9,10]; can the improvement over time be related to the enhanced skill achieved through practice? Did the pattern of patient referral to the study hospital change, perhaps due to the fame acquired from their academic interest? Could improved diagnostic screening in the community have led to earlier referral and less need for extensive surgical dissection? Conversely, did changing members in the house staff, intensive care unit, or surgical teams contribute to the measured outcomes? Surrogate measures such as duration of the operative procedure or extent of blood loss are often recorded to reflect the extent of trauma, yet they are unable to describe all aspects of the process of patient selection, preparation, or perioperative course. Such critique notwithstanding, the goal of observational studies is the description of the course of management most likely to achieve the best result. It is seldom known which specific aspect of the new protocol actually caused the improved outcome, because multiple variables are implemented at once. That, however, is not important: the authors are assessing a system of care, not the dissected individual components. Both observational studies and RCTs are subject to the possibility that participation itself somehow influenced the final outcome [the Hawthorne effect [11]]: a before and after evaluation may be positively affected by closer attention to the recent subjects. However, observational studies occur in the actual community of health care, are conducted by those who deliver service, and are applied to an unselected group of subjects. Moreover, the observers are not blinded to the patient assessments or to the treatments, which allows for the individualization of care. Their outcome measures, collectively referred to as effectiveness, are held by many [12] to be more indicative of the success of the health care system than the efficacy measures arising from the artificial environment of laboratories or teaching hospitals. Although one should limit interpretation of observational results to drawing associations rather than to generating proof, the value of well conducted observations in generating new concepts and ideas for future RCTs cannot be underestimated [13]. A weakness of many RCTs is the lack of sufficient size to demonstrate a true outcome when in fact one does exist; lack of statistical power has lead to combining several small RCTs with similar findings on the same question, so-called meta-analysis [14]. In doing so, it is assumed that randomization eliminates bias and confounding variables, both known and unknown, and the augmented power of greater numbers will address the potential type II error for effects of low magnitude. In nonexperimental studies, it is impossible to rule out bias introduced by lack of blinding and confounding variables; if the same systematic biases are present across a number of observational studies, meta-analysis will reinforce them and produce spurious statistical stability [15,16]. The limits of discerning power of observational studies is therefore set by the study design itself; the patient numbers from different trials cannot be combined in some quasi-experimental fashion to achieve significance. It takes a plausible hypothesis, repeated studies of the same association, and examination in various settings before a causal argument [17] can be established for a given treatment and a specific outcome. With respect to Brodner et al.'s work, one must conclude that the application of multimodal pain therapy is attractive, but others must come to similar conclusions before results can be considered broadly generalizable. Health services research must be based on the principles of good science [18]: a sound hypothesis, a considered methodology with appropriate controls [19], and careful observations with appropriate statistical analysis [20]. The inability to prospectively manipulate variables or to randomize subjects places a greater burden on the authors to demonstrate that the comparison groups are acceptable and that outcome differences are attributable to the hypothesis rather than to some undefined variable. The reader's decision whether to incorporate a study's findings into one's practice is dependent on the methodology, the magnitude of the outcome effect, and the uniformity with which a given association has been found by different authors in different circumstances [21,22] (Table 2). Level I studies can lead to definite clinical recommendations, and level II studies may be combined in an appropriate meta-analysis to increase statistical power. In contrast, it would be prudent to consider that any recommendation arising from level III, IV, or V evidence be one of an array of potential clinical actions, any of which would be considered appropriate under similar circumstances.Table 2: Levels of EvidenceaObservational research may seem an imprecise science but is achieving greater importance as the pace of medical change accelerates; RCTs cannot quickly answer questions about sweeping changes to medical systems. Indeed, the strength of each research design addresses the weakness of others! Anesthesiologists need both types of studies to support evidence-based practice and to explore new concepts and ideas. We can learn from observing change, but with limitations acknowledged.