Can systematic reviews with sparse data be trusted?

Systematic reviews are increasingly used to answer clinical questions. Their popularity is partly attributable to their apparent clinical superiority in providing quick summaries of evidence from the ever-growing medical literature (more than 750,000 items annually 1), but also thanks to the great impact of the Cochrane Collaboration in promoting the use of the systematic review 2. We also note that many scientific journals favour systematic reviews because they tend to contribute to higher journal impact factors. However, the rise of systematic reviews should give cause for concern; more than 8000 are being indexed in MEDLINE annually, reflecting a three-fold increase over the last decade 3. We write ‘concern’, because many of these reviews across healthcare seem to fall short of essential requirements such as incorporating risk of bias assessment in meta-analyses (only 16%), searching for unpublished data (only 7%), or not fully following a protocol (only one-third 3). A recent study of reviews published in anaesthetic journals revealed similar shortfalls 4. Further more, the quality of reporting is lower in some countries and in journals with lower impact factors 5, 6. The apparent poor quality of many published systematic reviews greatly increases the risk of misleading conclusions, which ultimately could harm our patients 7. Despite their veneer of sophistication, one has to bear in mind that systematic reviews are, in essence, retrospective appraisals of the evidence. Editors, writers and readers should exercise great caution when interpreting conclusions of reviews that only include very few trials, even when they fulfill the required standards 3. Despite great efforts to promote adequate quality of reviews through international guidelines such as PRISMA, GRADE and the Cochrane Handbook 8, the majority of reviews may suffer from threats to the validity of their findings due to risk of systematic errors (bias), design errors (i.e., erroneous selection of patients, doses of medication, comparators, analyses, outcomes) and random error (play of chance) 9. In a previous publication in this journal, some of the methodological issues and requirements for authors and editors were thoroughly discussed 10. But we believe that other issues need to be addressed, in particular the amount of information included in meta-analyses from systematic reviews, with the power this brings. One often-debated question is the relative superiority of evidence derived from several small trials versus that from one large trial. Some have claimed that evidence produced in a large randomised controlled trial (RCT) is more valuable than that from a systematic review 11. However, in our opinion, evidence from high-quality systematic reviews is intuitively superior to that of one large randomised controlled trial, if the authors acknowledge and take into account the inherent methodological limitations of the systematic review and address these adequately 12, 13. Another recent approach that is becoming increasingly popular in cases of sparse data is the use of network meta-analysis, which permits both direct and indirect head-to-head comparison of interventions from various RCTs 14. However, despite its apparent advantages, it remains a complicated approach, requiring a thorough protocol and clear hypothesis in order to avoid incoherence and falsely positive findings, as a high number of pairwise comparisons might ultimately increase the risk of type-1 error. Implementation of the new PRISMA guideline for network meta-analyses should, in our opinion, be mandatory 15. But how should authors and journals deal with systematic reviews that fail to include more than a few small trials, often with a high risk of bias? Before addressing this question, one has to understand the inherent threats to the internal validity of the findings of such publications. First of all, trials with a high risk of bias lead to biased intervention-effect estimates; certainly, smaller trials tend to overestimate intervention benefits and underestimate harms 16. There is abundant empirical evidence supporting the fact that bias associated with trial design characteristics such as inadequate or unclear random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data reporting, selective outcome reporting, lack of trial protocol registration and vested interests (academic or industry driven), will significantly influence the reported findings of trials, often overestimating the intervention effect significantly 9. Thus, systematic reviews of only a few small trials will often perpetuate the shortcomings of these biased trials. Equally, trial design errors, such as abuse of surrogate outcomes with dubious clinical relevance, may lead to the introduction of interventions that are neither meaningful nor safe for patients 17. Ideally, a core set of outcomes should be developed in accordance with the COMET (Core Outcome Measures in Effectiveness Trials) initiative (http://www.comet-initiative.org) 18, which would provide more appropriate and patient-centred research. Within peri-operative medicine, some moves have already been made in this direction 19. But unfortunately, the reality is that a great proportion of published RCTs still fall short of including outcomes of relevance for patients, clinicians and policy makers 20. Again, this will naturally affect the conclusions of reviews that include these trials. Another increasingly used approach in RCTs and systematic reviews with sparse data is the use of composite outcomes, since this reduces the required sample size 21. However, this strategy not only creates difficulties for readers trying to interpret the clinical significance of the findings, but may also provide false answers 22. Ideally, composite outcomes should only be used if the authors equally provide data on each component of the analysis so the readers are able to interpret the clinical implications of the results 17. Inclusion of non-inferiority trials in systematic reviews is equally problematic, since one could argue that the non-inferiority design in reality should not even be permitted, as new interventions ideally have to be superior to existing ones and not put patients at risk of harm. However, this approach is often preferred since it – once again- reduces the required sample size, and thus can help industry introduce new interventions 23. Furthermore, when interpreting results of systematic reviews with sparse data, selective outcome reporting and publication bias are a major source of unreliability; trials with positive findings are more often published than trials with neutral or negative findings 24. This will lead to an increased risk of random error (play of chance) and an increased risk of false conclusions about interventions 25. This has led to campaigns for the publication of the results of all trials, such as the AllTrials initiative (http://www.alltrialsnet/) and the WHO initiative for public disclosure of all trials (http://www.who.int/ictrp/results/WHO_Statement_results_reporting_clinical_trials.pdf). But selective outcome reporting also appears to be present in many systematic reviews; authors should not only be obliged to publish their protocols in advance, but editors should assess adherence to these protocols in order to minimise these post-hoc decisions that often are based on observed results and their direction 26. Cochrane reviews begin with a published protocol; these may also be made publically available in journals such as Trials, Systematic Reviews, JBI Database of Systematic Reviews and Implementation Reports, or registered with PROSPERO. The cumulative sample size remains a vital element of reliability for any result of a meta-analysis from a systematic review, as is the case for any RCT. Inadequate sample size may result in not only type-1 errors (false positive conclusions) but also type-2 errors (false negative conclusions). Systematic reviews which include only a small number of trials, or few patients, in their meta-analyses, will thus have an increased risk of false conclusions. The degree of uncertainty will only increase further as more statistical tests such as sub-group and sensitivity analyses are carried out in these reviews. The precision and reliability of the findings of such reviews will further be hampered in the case of substantial inter-study heterogeneity (i.e., methods, population, interventions, duration of intervention) or other issues such as variation across trials 27. As a consequence, it is essential for editors and authors to address the issue of the cumulative sample size (the information size) in reviews. But, nevertheless, this issue remains neglected 17, and it remains unusual for review authors to perform such calculations when findings are negative 28. In fact, almost 80% of Cochrane systematic reviews (perceived to be methodologically superior to other published systematic reviews) are underpowered to detect or reject a 30% relative risk reduction 29. Further more, 97% of all Cochrane review meta-analyses did not reach 80% power to detect or reject a 10% relative risk reduction 29. Additionally, in our own specialty, the majority of systematic reviews examining interventions which are reported as statistically significant appear to have problems with power (information size); as few as 12% of meta-analyses have a power of ≥80%, while inadequate precision and type-1 error constitute another major source of concern 30. Finally, sparse systematic reviews are more likely to be updated in the future, which results in a sequential multiplicity that increases the risk of random error 31. Unfortunately, this issue is also often overlooked in most scientific journals. As a consequence, the majority of meta-analyses in systematic reviews should be considered as nothing more than interim analyses in a continuing (and often lengthy) process of reaching the required information size 32. To combat this problem, various sequential approaches have been proposed, such as trial sequential analysis 33, 34, Whitehead's triangular test 35, the law of iterated logarithm 36, and Bayesian methods 37. In general, these approaches provide a more conservative threshold for statistical significance for systematic reviews with meta-analyses of small trials or few participants. Trial sequential analysis offers various advantages. It combines conventional meta-analysis with thresholds for declaring significance in cases of sparse data and repeated updates. It provides information on the required information size (power) but additionally maintains the overall risk of type-1 error ≤5%. Additionally, the required information size calculated by this method includes an estimate of heterogeneity, including the possibility that between-trial variance could be a result of random error and not only due to systematic differences. Thus, trial sequential analysis can enable us to assess when firm evidence is reached in meta-analysis and provide valuable information on the lack of, or presence of, clinical effect as early as possible 38, 39. But, just as importantly, where systematic reviews have only a few trials, it will often render apparently statistically conclusive meta-analyses inconclusive due to lack of power 40. Finally, it is possible to use this approach to calculate the required number of future trials needed to ‘plug the information gap’ 41. Despite adherence to best practice guidelines on how to write systematic reviews and despite application of Cochrane methodology, systematic reviews with sparse data pose a great challenge for patients and policy makers due to their inherent unreliability and imprecision, since the estimated proportion of false-positive meta-analyses may be as high as 37% 42. We suggest that editors and review authors should not only work to PRISMA, GRADE and the Cochrane Handbook but also think about information size in reviews 43. The conduct and reporting of under-powered, single-centre, low quality and biased trials continues because academic institutions and physicians are still driven to publish 44. Such trials taint the systematic reviews which include them. Reviews are also hampered by the large number of unpublished trials (possibly nearly half 45) and the presence of selective outcome reporting 46 and even misconduct and fraud 47; journal peer review does not always seem to protect the scientific community against such activity 44. So how should we tackle the problem of systematic reviews with sparse data? Certainly to begin with, it should be mandatory for authors to register review protocols and adhere to certain international standards (i.e., Cochrane methodology, PRISMA guideline, GRADE) before journals even consider systematic reviews for publication. We strongly advocate that journals and editors demand the use of sequential methods for calculation of the required information size based on a plausible intervention effect in order to reduce the risk of false positive findings and conclusions. And finally, as advocated by others, we suggest that the time has come to put more emphasis on trials that have been prospectively registered and on trial registries which include full protocols and datasets before trials are heavily weighted in systematic reviews 44. AS is an editor of Anaesthesia. No other external funding or competing interests declared. Although the views expressed in this editorial are those of the authors, the editorial policy of Anaesthesia will be revised in line with some of the suggestions made. Thus, we would urge authors of systematic reviews with sparse data to ask themselves whether such reviews should even be submitted for publication, and consider whether trial sequential analysis might usefully be employed if so. We already encourage registration of systematic reviews before work begins. Those reporting network meta-analyses will be required to follow the relevant PRISMA reporting guidelines.