Systematic reviews in public health: old chestnuts and new challenges

The Bulletin’s editorial team has decided to encourage systematic reviews. This is an important decision and is to be welcomed. However it is possible that it will not be received positively by all. Some may feel that systematic reviews are not suitable for answering wider public health questions, and that the new section will simply result in the publication of systematic reviews addressing narrowly focused, clinical questions with a focus on randomized controlled trials alone.1 There is no reason why this should be so, and there are far too many examples of systematic reviews of non-clinical interventions (including government policies), which include non-randomized studies, to list.2 Systematic reviews of qualitative research are also becoming more common.3 Nonetheless misconceptions about the scope and flexibility of systematic reviews still surface occasionally and it never fails to surprise me when I stumble across one of these hoary old chestnuts. If it really were true that systematic reviews were only for simple, individual-level interventions, systematic reviewers working in public health would be very concerned. After all, public health problems require us to draw on complex sets of qualitative as well as quantitative evidence; and some policy interventions have never been subjected to randomized controlled trials (RCTs) but have been evaluated using other approaches; everything from controlled and uncontrolled before-and-after studies, to time series analyses, to qualitative methods. In such cases, systematic reviews that focused on RCTs alone would have little to say and we would end up producing many “empty reviews”, which contain no studies at all. These would be undoubtedly quicker to do and quicker to read than the typical systematic review, but would be of limited use for making policy. However this is not the case. In public health we already have a wealth of reviews that prioritize RCTs where they are available, but which draw upon a range of other evidence where they are appropriate to answering the review question.2 Take, as an example, transport policies, an important social determinant of health.4 Public health decision-makers have been interested in the effects on health of building new roads for decades, but there has never been a randomized controlled trial, for obvious reasons. Yet it would make no sense to claim that there is no evidence to be reviewed; rather, there have been many controlled and uncontrolled studies that have examined the effects of new roads on respiratory symptoms, injuries, community severance, disturbance and a variety of other outcomes of public health importance. These studies have in fact been systematically reviewed.5 Many other examples of reviews of complex interventions that impact on public health could be given, which appear on the Cochrane Database and elsewhere,6,7 and such reviews are the main focus of the recently-established Cochrane Public Health Review Group. It is also worth noting that both the Cochrane and Campbell Collaboration include increasing numbers of reviews that are relevant to low- and middle- income countries,8 though there is a particular need for more systematic reviews that consider the effects on equity of the interventions that they include.9 Currently systematic reviews have a “utilitarian bias” – they tend to be concerned more with the effects on populations and average effects than with distributional effects and impacts in disadvantaged subgroups. It is to be hoped that systematic reviews submitted to the Bulletin will routinely consider the effects of interventions on health inequities. The Cochrane Health Equity Field (http://equity.cochrane.org/en/index.html) was set up explicitly to further this agenda, to encourage systematic review authors to explicitly assess the effects of interventions not only on the whole population, but on the disadvantaged. So where does this leave non-systematic reviews? Should every review be a systematic review? This is a difficult question to answer in a short editorial. My own view is that not all reviews need to be systematic reviews; there is still a place for review papers that provide a broad overview, and that discuss a range of evidence and make a contribution to wider debates about what might work in particular settings. However, while such discussions help place evidence in context and suggest how it may be used, they should not be confused with the evidence-based conclusions of a transparent, scientific review, which systematically attempts to find all the relevant evidence, appraises its strengths and limitations in a transparent fashion, and draws that evidence together in such a way that the more robust studies are given greater weight. Finally, the challenges are many. Evidence on interventions to improve public health is in short supply, particularly evidence on social determinants. Systematic reviewers therefore face a real challenge in making best use of the sometimes poor, often sparse and usually heterogeneous evidence available to them. It is easy to set narrow inclusion criteria for a review and then sift through the evidence before concluding that there isn’t enough – and what there is, isn’t very good anyway. Such “evidential nihilism” is probably not helpful to decision-makers, and public health systematic reviewers need to continue to develop new methods and better frameworks within which different types of research evidence can be integrated to inform decision-making. Publishing systematic reviews in the Bulletin should give an added impetus to this ongoing work. ■