Systematic reviews: much ado about a lot

In this issue, Edwards and colleagues [1] provide us with a cogent overview of what goes into crafting a systematic review of randomised controlled trials. By indirectly comparing different interventions, we can estimate potentially important differences between treatments that otherwise have not been tested, nor may never be tested, in a single study. Systematic reviews of randomised trials sit atop the hierarchy of strength of evidence for treatment decisions, bested only by the often mythical N of one randomised clinical trial [2]. Systematic reviews of randomised trials are generally considered superior to the usual single randomised trial or to observational studies. It behooves the clinician to understand what meta-analyses can and can’t tell us. Together with a prior review published in this journal [3], the practising clinician is better prepared to sort out competing claims for the myriad of interventions being actively promoted. A critical eye is still required as exemplified by recent controversies centring on bias [4]. For example, when constructing a meta-analysis, some trials are included and some are not. Bias may be introduced when negative studies are not included, perhaps because they are not always published. This is becoming less of a problem now with the availability of on-line clinical trial registries such as http://www.clinicaltrials.gov, and the release of briefing documents by drug regulatory agencies such as the US Food and Drug Administration. This so-called ‘grey literature’ of unpublished and difficult-to-retrieve studies may tip the balance of a meta-analysis [5]. For example, consider the controversy regarding the efficacy and safety of antidepressants in children and adolescents. In a systematic review that included the grey literature, it was observed that published data suggested a favourable risk–benefit profile for some SSRIs; however, addition of unpublished data revealed that risks could outweigh benefits for some of these medications [6]. On the other hand, the grey literature may at times be devoid of sufficient detail to assess study quality and also has not undergone the peer-review process expected of a published study report in a reputable journal. Another potential pitfall with meta-analyses of randomised clinical trials is that there may be a mismatch between the type of patients enrolled in clinical trials vs. the patients one commonly encounters in practice, with the latter often burdened with a more severe level of symptomatology or with a variety of comorbidities that would have excluded them from participating in a randomised clinical trial. This is particularly a problem for the interpretation of drug registration trials. As a possible solution, Edwards and colleagues [1] discuss a methodology that can also incorporate data from observational studies into a systematic review. Applying the results of a meta-analysis to one’s clinical practice entails verifying that the outcomes that were measured are relevant to you and your patients. Other questions to ask include: would your patient have been eligible for any of the studies included in the meta-analysis? Is the dose of the medication that was studied similar to the dose you are contemplating? Is the measured outcome, such as reduction of symptoms below a set threshold, commonly achievable? The magnitude of the treatment effect must be clinically significant. Perhaps the most intuitive effect size measure is number needed to treat (NNT), a measure often reported in meta-analyses, and that can be communicated effectively among clinicians and with patients [5,7]. Greater numbers of meta-analyses are finding their way into medical journals [8]. Indeed, an emerging topic is the meta-analysis of meta-analyses (a meta-meta-analysis) [9]. Part of the explosion of interest has been the automation of statistical testing. The ‘heavy lifting’ is no longer the calculation of effect sizes and confidence intervals, but the searching for, acquisition of, and inputting of data from relevant clinical trials. The Cochrane Collaboration, a global network of researchers maintains an informative website (http://www.cochrane.org/), and distributes a handbook that describes in detail the process of preparing systematic reviews of healthcare interventions. Software called ‘Review Manager’ produces the attractive graphics that readers of Cochrane reviews are familiar with. Searching for a particular abstract or summary can be carried out at http://www.cochrane.org/reviews/. It’s worth a look. Leslie Citrome, is a consultant for, has received honoraria from, or has conducted clinical research supported by the following: Abbott Laboratories, AstraZeneca Pharmaceuticals, Avanir Pharmaceuticals, Azur Pharma Inc, Barr Laboratories, Bristol-Myers Squibb, Eli Lilly and Company, Forest Research Institute, GlaxoSmithKline, Janssen Pharmaceuticals, Jazz Pharmaceuticals, Pfizer Inc, and Vanda Pharmaceuticals.