Reviews, systematic reviews and Anaesthesia

Doubt is not a pleasant condition, but certainty is an absurd one. – Voltaire 1 Last year, Anaesthesia published 24 review articles. A glance at the Journal covers shows that these varied not only in subject matter, but also in scope and methodology. Of the 24 articles, 11 were systematic reviews with meta-analysis (e.g. 2, 3), one was a systematic review with meta-regression, and two were systematic reviews without statistical combination of data 4, 5, one of which was described as ‘qualitative’ 4. The remaining 13 were what one would describe as ‘narrative reviews’ (for instance 6, 7); four of these 8-11 made some attempt to outline methodological features such as reporting date limits for literature searching, assessing the methodological quality of retrieved material, or reporting the numbers of papers found and included. Review articles are popular with readers and researchers as, at their best, they provide up-to-date summaries of published research and/or answers to practical clinical questions. Traditional, ‘narrative’ reviews, typically where an expert in the field is invited to write a summary of what is known, may include a more selective sample of the available literature and have been thought to be more open to bias. Systematic reviews, on the other hand, follow certain pre-specified steps in formulating a question, searching the literature, deciding on material for inclusion and extracting data; the objectivity and replicability of these steps help to reduce bias in the finished review. They may or may not include a numerical summary of the primary trial data (meta-analysis). Systematic reviews have grown in popularity recently, helped by the growth and influence of the Cochrane Collaboration 12. Within the Collaboration, there is a Cochrane Anaesthesia Review Group which deals with anaesthesia, critical care, and peri-operative and emergency medicine. Last year, the Group published 34 new reviews and nine updated reviews. Again, the topics were similarly diverse, ranging from the role of anaesthetics in tumour recurrence 13, through protective ventilation in acute respiratory distress syndrome 14 and peri-operative beta-blockade 15, to the prevention of hypothermia 16. Despite their growing prevalence and apparent scientific superiority, many readers find systematic reviews difficult. There are indeed some statistical features that may be unfamiliar, complicated by the fact that many methodological issues are controversial or still under development. However, the most important point is that, despite their appearance of objectivity, systematic reviews are, in essence, retrospective, observational studies and can never achieve the same level of reliability as a large, well-conducted primary study of the same intervention. So, even if great attention has been paid to the various methodological aspects (see below), there will always be limits to what can be inferred from the review. Readers (and reviewers) should not lose sight of this simple fact, and remember that certainty is unlikely – even if not, as Voltaire suggested, absurd. A useful way of thinking about reviews is to compare the methods with those used in primary clinical trials. The randomised controlled trial (RCT) has achieved cultural supremacy as the least biased means of establishing the effects of interventions. Most doctors are familiar with the idea that randomisation and blinding will reduce bias; the nuances of this are discussed further below. It also makes sense to handle data from patients who started the study, but did not complete it, in a way that does not skew the results. Usually this is achieved by ‘intention-to-treat’ analysis. Finally, it is not uncommon for authors to report some outcomes, but not others – so-called ‘selective outcome reporting’. Usually this is because the study found an effect on some outcomes but not others. If all the proposed outcomes were specified at the beginning of the paper, it is easy to identify the omission. If, however, they do not appear in the published report at all, this is not possible. The solution to this is to publish trial protocols before recruitment begins. There are many trial registers for doing this, and Anaesthesia expects that researchers register their trials if they plan to publish in this journal. Analogous biases can beset systematic reviews. For instance, incomplete searching for literature can skew the available pool of material. Authors may choose to filter their search to exclude papers in languages other than English. Publication bias – where studies that demonstrate an effect from an intervention are more likely to be published than studies that do not demonstrate an effect – may be more likely if a search for unpublished material, or studies published only as abstracts, is not carried out. Decisions on which papers to include, and which to leave out, will inevitably be partly subjective, but the process should at the very least be reproducible and reported in sufficient detail. Finally, decisions about quality thresholds for inclusion in analysis are important; poor-quality studies are more likely to overestimate the effects of interventions. So far, so good. But there are many other features that are apparently not well understood, even by authors who submit systematic reviews, either because journals’ methodological standards and expectations have changed over time or because they are inherently complex. We will now attempt to explain more about reviews, with a view to helping readers understand more about them, and at the same time setting out what Anaesthesia expects from systematic reviews submitted in the future. We shall refer to the most recent systematic review to appear in print in the Journal, which demonstrates some of the features described below 17. First, just as with a primary research project, it is important to plan the work properly. If the protocol is thorough, the chances that the review might misrepresent the evidence will be minimised. Often, reviewers do not follow their protocol (whether through acts of omission or commission), or the protocol was inadequate, or indeed never existed. Ideally, a protocol should also be registered, for instance with PROSPERO (see http://www.crd.york.ac.uk/PROSPERO). Authors should be able to state concisely the objective of their systematic review, structured as a ‘PICO’ question (specifies the population studied, the intervention used, any comparator and the outcomes of interest). Authors need to make a judgement too about how broadly the limits for the study population and intervention(s) should be set. Some would argue that larger, inclusive reviews should be preferred to restricted, exclusive reviews; for instance, there is no evidence that many anaesthetic interventions exert different effects in different types of surgery. Further, reviews that are too restricted and exclude a lot of studies will have less power and precision, and produce higher rates of spurious findings. In terms of inclusion and exclusion criteria, similar principles apply. Authors often forget that they are including summary data from trials rather than individual patient data, which makes criteria such as ‘exclude patients less than 18 years old’ inapplicable because, in this example, trials might summarise results from both children and adults. Authors should also not exclude trials because of differences in populations rather than proven differences in the effect of an intervention. For instance, in one trial an antiemetic might reduce vomiting from 80% in a control group to 60% and in another trial from 40% in a control group to 30%: the absolute reduction in the first trial was 20% and in the second trial it was 10%, but the relative effect of the drug was the same, a relative rate reduction of 0.25. Authors might be tempted to analyse these two trials separately, for instance if the first was in children and the second trial was in adults, in the mistaken belief that the difference in absolute effects represents an interaction between participants’ age and drug effect. An unbiased and exhaustive search for material is next. The search for intervention RCTs should always include CENTRAL, MEDLINE and EMBASE; regional databases, such as LILACS (see http://lilacs.bvsalud.org/en/) may also be useful. Google often reveals unexpected papers. The reference lists of included papers should also be scrutinised, as additional articles may be discovered. Making sense of the quality of studies can be tricky. Risks of bias should be assessed separately within recognised methodological domains and should be summarised in a table that illustrates the risk categorisation for each RCT (e.g. see Fig. 2 in Corredor et al.'s review 17). Summative scores are now considered to be unacceptable, thus: “One commonly-used scale was developed by Jadad and colleagues for randomised trials in pain research [18]. The use of this scale is explicitly discouraged. As well as suffering from the generic problems of scales, it has a strong emphasis on reporting rather than conduct, and does not cover one of the most important potential biases in randomized trials, namely allocation concealment” 19. Also, quality assessment on its own is only half the story. Many authors enter all the studies into the analysis no matter what the quality, but it is known that poor-quality studies tend to overestimate the effect of an intervention and this may distort the overall findings 20. One of the purposes of sensitivity analyses is to assess how sensitive are the results to the assumptions that led to the inclusion and exclusion of trials, including their reported methodological rigour. (This and other commonly-used terms are defined in Table 1). Further, authors should clearly distinguish – in their own minds – the difference between the generation and the concealment of an allocation sequence (both of which are covered by the word ‘randomisation’), and attempts to make the control and interventions indistinguishable (‘blinding’). Authors often fail to appreciate that a random sequence does not mean that the sequence was concealed; a random sequence could potentially be displayed for all to see, in which case it has little power to limit selection bias. Performance of the study can be biased if the control and intervention can be distinguished, for instance if they look, feel, smell or taste different to each other. Even if they are indistinguishable, one might still be able to distinguish between patients receiving an intervention and those receiving a control if their allocation sequence has been revealed, or if effects other than that being assessed are obvious (for instance, the tachycardia caused by cyclizine in a trial of antiemetics). There are usually more than two people in a study who can introduce bias, so the term ‘double-blind’ is meaningless; most studies should be ‘quadruple-blinded’ at least!1 In addition, the magnitude of bias that can be introduced is likely to differ between people in a study. Therefore, the likely success of blinding should be assessed separately for participants, for different personnel caring for the participants, and for the assessors who are identifying the presence or magnitude of an outcome. In addition, there may be subject-specific methodological issues. It is becoming increasingly common for authors to perform funnel plots to explore ‘small studies effects’. Readers will be familiar with funnel plots of hospital performance, for instance mortality after different surgeries, and whether their hospital is an ‘outlier’. The funnels in systematic reviews plot intervention effect, for instance relative mortality rate (rather than mortality in the hospital example), against some measure of statistical confidence in that rate, for instance the standard error (rather than the number of operations in the hospital example). Sometimes the funnel looks less like a symmetrical Christmas tree and more like one missing its lower branches. Authors often interpret such funnel plot asymmetry as the result of publication bias, but this is only one of at least seven reasons why a funnel plot might be asymmetrical. Conversely, publication bias can be consistent with a symmetrical funnel plot, in that extreme results ‘for’ and ‘against’ an intervention are published, whilst studies that do not show an effect remain unpublished, leaving the bole of the Christmas tree denuded. Funnel plot asymmetry should be determined statistically, using regression analyses such as Egger's, Harbord or arcsine analyses, not by eye. In the event of statistical evidence of asymmetry, a modified Galbraith plot – or similar – should be supplied to illustrate it 22. All tests lack power, so the possibility of small studies effects should be considered even in the absence of statistical proof: causes for funnel plot asymmetry can be explored through sensitivity analyses. Conversely, some measures of effect, such as the log odds ratio, can generate asymmetric plots even when small study effects are absent, as they are mathematically linked to their error. Subgroup analyses can be used, if necessary, to explore associations with the type of surgery or other factors – but they should be used sparingly, preferably following a-priori rationales published in the protocol. Typically, just as in a primary study, there are too many subgroup analyses in a review. An alternative method, meta-regression, may also be used to explore the interaction between a dependent outcome and explanatory variables. Neither analysis should be pursued if there are fewer than 10 studies for every variable explored. Sensitivity analyses should explore how the results are altered by justified changes to the inclusion or exclusion criteria and with changes to the method of analysis. The calculation of heterogeneity is straightforward within commonly-used software. (This should be expressed using the I2 statistic as a default). However, heterogeneity is poorly understood and often not explored, even when the I2 result suggests that this is warranted. Heterogeneity can be explored by assessing whether it can be reduced by subgrouping trials on the basis of biological or experimental rationale, preferably predetermined in the protocol. The most powerful method of analysing the association of factors with the effect of an intervention, and the heterogeneity of that effect, is through independent patient data, i.e. analyses at the patient level, rather than the trial level. This pursuit will probably only be viable for a meta-analysis of a novel intervention, since individual patient data may become unavailable if they are not stored securely. More detail and useful advice on this and many other methodological issues can be found in the Cochrane Handbook 19. Further help in interpreting systematic reviews is available: one of us (JC) published a ‘how to’ guide in Anaesthesia in 2007 23. This dealt with numerical presentation of both dichotomous and continuous data in review, explained how to interpret both funnel plots and forest plots, and outlined the role and meaning of subgroup and sensitivity analyses within reviews. In the last decade, methods have become available for meta-analyses of trials not only of interventions, but also of diagnostic tests 24, prognostic tools 25 and non-randomised studies 26. Also relatively new is network meta-analysis 27, which allows the evaluation of the effect of an intervention among similar patient populations that have not been compared directly in the same clinical trial 28. There are even overviews of reviews 29. There is also some interest in trial sequential analysis, a technique that can serve the same purpose as power calculation and interim data monitoring in a primary clinical trial 30, 31. It allows the calculation of the ‘information size’ (number of patients) that would have to be included in a meta-analysis to demonstrate an effect definitively. Such sequential methods can provide information on when firm evidence is reached in a cumulative meta-analysis, and can also provide an early clue to the lack of presence of a clinical effect. Finally, the systematic review can cast light on other aspects of practice; within Anaesthesia we have recently published a systematic review with meta-regression 32, a review of the interaction between ondansetron and tramadol 33, and case reports and series of cranial nerve injuries with supraglottic airways 34. Manipulation of the existing dataset from a systematic review also helped confirm a recent case of research fraud 35. Finally, to accompany the PRISMA statement for the reporting of finished reviews, there is now PRISMA-P too, a template to guide the reporting of review protocols 36. Systematic reviews are complex and may suffer from bias and lack of adequate power, and, just as for primary clinical trials, complete certainty may be an absurdity. As such, evidence and opinion both have their place in science and medicine, and hence so do both narrative and systematic reviews. We continue to invite authors to submit them both to Anaesthesia. Further guidance on how these should be conducted and presented is now in the ‘Author Guidelines’ on the journal's website (see http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1365-2044/homepage/ForAuthors.html). Both authors are editors of Anaesthesia. AFS is an editor in the Cochrane Anaesthesia Review Group, as was JC from 2002 until 2014. AFS is also the recipient of a Cochrane Collaboration programme grant from the UK National Institute for Health Research.