Systematic reviews - A perspective on benefits and concerns in 2011. Safe, sound and sorted?

Clinicians require concise accurate information on which to base healthcare decisions. The route by which this has been provided in the last 10 years has been changing. There has been an explosion of accessible medical literature on the internet and many journals now publish papers months or even years before they appear on paper. Clinicians face two dissonant problems. First, only a limited amount of clinical papers are available for free access and secondly, the volume of data is increasing exponentially. For example, using only PubMed (simple and free), there are around 2000 new citations for schizophrenia in the last 2 years. As a result, clinicians often turn to review papers on a single relevant topic. Simple review papers cover a multitude of sins with personal opinions and quality ranging widely. Formal, systematic review, however, offering a complete synthesis of a topic, is not only a superior methodology but should remove some of the biases inherent albeit unintended, within an unstructured review. Not surprisingly there has been an exponential increase in the number of cited systematic reviews in Medline from less than 500 in 1994 to more than 5000 in 2009 (1) and these data are likely to be an underestimate (2). This increase has led commentators and reviewers to suggest that not only should the quality of these systematic reviews be improved, there should also be a uniformly agreed structure for systematic reviews along similar lines to those adopted to improve the quality of reporting of meta-analyses in 1996 (3,4). In 2009, the PRISMA statement was published listing preferred reporting terms for systematic reviews and meta-analyses (4). This represented a major advance in that the previous QUORUM statement focused solely on reducing the sub-optimal reporting of meta-analyses (3). The PRISMA statement is a 27-item checklist and the authors have also helpfully produced a downloadable template and flow diagram for all (4). In addition, the authors make a critical learning point that is not always recognised, that systematic review is hierarchically the stronger methodological tool and may or may not include a meta-analysis (4). Put concisely, until systematic review of the data has taken place, there is no opportunity to evaluate the correctness of performing meta-analysis, which would hence integrate the results of the included studies. Often meta-analysis is inappropriate (5,6) for any number of reasons. Some would argue that the questions posed within a systematic review should allow a protocol to be developed prior to undertaking the review. Currently only 10% have such a protocol (2) but curiously, in 2004, only 50% of systematic reviews even included the terms ‘systematic review, meta-analysis’ in the title (2). The same analysis also reported that funding sources were declared in only 40% of reviews (2) which may represent a critical determining factor in their usage. Advantages might include external critical appraisal of the project prior to its undertaking if there was a uniform registry of such protocols. An international registry of systematic review protocols has not only been proposed but also it recently underwent an open consultation process with opinions from international experts in different arenas (including journal editors but not specifically the pharmaceutical industry) sought (1). The registry will be produced and maintained by the Centre for Reviews and Dissemination and will remain open-access; it will link resulting publications but will initially at least be limited to systematic reviews of the effectiveness of health interventions in the arena of health and social care. Others may follow and indeed, in the UK, there is already a pilot for an All Wales Systematic Reviews Register (1). All the above and more leads to calls for systematic reviews to fundamentally replace ‘reviews’ of an undefined nature. Frankly there is no real argument against this. Systematic review is by itself a learning process. Some of the learnings by the author from his three (and soon to be four) systematic reviews have been seminal. For example, a systematic review of usage of metformin for weight management or loss in antipsychotic treated patients reported on 11 studies in almost 500 patients, and in general terms offered encouraging data (5). However, a significant limitation was that no Caucasian patients participated in these trials, (ethnicity is likely to play a role in metabolic effects of drugs) (7) thus limiting generalisation of any conclusions and the potential relationship between benefits ascribed to metformin and concurrently used weight management programmes was tangled and complex. Our most recent systematic review looked at the incidence of breast cancer in females patients with schizophrenia and concluded it was likely to be significantly increased (of concern as breast cancer has a lifetime incidence of at least one in nine women) (6). We reported widely discrepant trial results ranging from a 40% decrease in risk to a 52% increase. This broad range of outcomes related to small case numbers in some studies, short-term follow up but most critically, data from populations which were too young to be at substantial risk of breast cancer. Breast cancer is essentially an illness that is diagnosed after 50 years and all studies reporting data in that age group found significantly elevated rates (6). In both these systematic reviews (5,6), a significant number of studies also came from a single geographic region and it was not possible to gauge the degree of overlap of patient cohorts involved in many studies which may inadvertently result in double-counting of their data. Are there any caveats for undertaking systematic review? Maybe, some of the recent discussions relate to systematic reviews undertaken by national bodies or funded organisations and may frighten the average clinician into considering it something beyond their competency (1). The review of 11 papers is not beyond the scope of any determined clinician and indeed could be considered an essential skill teaching clinical appraisal (5). The median number of studies in a therapeutic systematic review is 16, including 1112 participants (2). The recent editorial in International Journal of Clinical Practice is correct (8), we should encourage systematic review as the norm and maybe it is time for journals to make clear the standards in their instructions for authors and even to include downloadable checklists (4). Lastly, the evidence from systematic review needs to be used by policymakers and clinicians to make a pragmatic difference in outcomes. A recent systematic review itself reviews the impact of systematic reviews on policymakers and does not report encouraging data (9). After 3 months of having received five relevant systematic reviews, only 23% of policymakers had utilised these in decision-making. Further research is needed into why data are not used, what data would be used but crucially giving good accessibility to the data via open-access publishing. Curiously in one of the randomised trials cited, having a ‘knowledge broker’, in addition to the data seemed to decrease usage of the systematic reviews (9). A separate systematic review evaluating usage of systematic reviews in clinical decision-making reported almost an absence of data (10). Systematic review is here to stay and standards will improve. All clinicians should be encouraged to consider it within their scope as we head for 2012. CB is an employee of Eli Lilly and Company Ltd.