Why the Cochrane Risk of Bias Tool Should Include Funding Source as a Standard Item

GP Kidd/Science Photo Library This editorial is based on a presentation given as part of a debate held at the Cochrane Methods Symposium in Québec City, Canada, October 2013. You can also read Jonathan Sterne's editorial presenting the case for not including funding source in the risk of bias tool. Rigorous systematic reviews aim to identify and reduce any bias that can adversely influence interpretation of study outcomes. The Cochrane Handbook defines bias as “a systematic error, or deviation from the truth, in results or inferences” and goes on to describe a number of biases, including allocation concealment and blinding, that have been shown empirically to systematically influence results.1 As Cochrane strives to improve its methods of bias assessment, we must consider the empirical data showing that funding source is a risk of bias. A recent Cochrane Review provides evidence that there is bias associated with study funding sources.2 The review examined the association of study funding source with clinical drug study outcomes, and the included studies defined ‘favorable’ results as those showing greater efficacy or less harm for the sponsor's product than for the comparator. Pharmaceutical industry‐sponsored studies were more likely to have favourable efficacy results (risk ratio 1.32, 95% confidence interval 1.21 to 1.44) and harm results (risk ratio 1.87, 95% confidence interval 1.54 to 2.27) than studies not sponsored by industry. The review found mixed results for effect size. This is completely expected since the effect sizes of different drugs can vary widely and depend on dose, the outcome being measured, and other factors. For example, the influence of reporting biases on effect sizes has varies considerably among drugs.3 Industry‐sponsored studies have favourable conclusions more often than non‐industry‐sponsored studies, even when controlling for effect sizes.4 Importantly, the consequence of industry funding is the same, independent of the effect size: industry‐funded trials show treatments to be more efficacious and less harmful than non‐industry funded trials. The current Cochrane risk of bias tool is insufficient to assess bias related to study funding sources. The Cochrane tool includes sequence generation, allocation sequence concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective outcome reporting. The Cochrane Review that examined funding bias found no difference between industry‐funded and non‐industry‐funded drug studies in sequence generation, concealment of allocation, or loss to follow‐up.2 Industry‐funded studies did have lower risk of bias related to blinding, but this is not surprising as many drug studies are conducted for regulatory approval and must adhere to certain study design standards. Furthermore, the lower risk of performance and detection bias in industry trials would be expected to lead to less, not more, favourable results. Since the observed funding bias cannot be fully explained by the study characteristics assessed with the Cochrane risk of bias tool, there must be other possible mechanisms by which this works. Bias related to funding source can results from systematic influences on how the study is actually conducted, the methodology of the study, whether the full results and analyses of the study are published, or a combination of these mechanisms.5 Drug study results can be biased to maximise efficacy and minimise harm through such mechanisms as choice of inferior comparators (either by dose, drug or administration route),6, 7 biased coding of outcomes,6 bias in how data are analysed,8 and selective outcome reporting and publication bias.9 Risks of bias are not mutually exclusive; a study may have multiple risks of bias and we may not be able to identify all of them. Funding source bias is a known bias that should be assessed. Furthermore, bias may be related to funding source even when all studies are industry‐funded. For head‐to‐head comparisons of statins with other drugs, funding bias is associated with which statin manufacturer funds the study, and not just industry funding per se. The head‐to‐head comparison was more likely to favour the sponsor's product than the competitor drug, even when other risks of bias in the studies were taken into account.10 So, when there is competition between drugs in a class, funding bias can influence the outcome to favour the commercial interests of the sponsor. If readers of Cochrane Reviews wanted to do their own assessment of bias related to funding source, they would find this difficult. Cochrane Reviews are not doing an adequate job of disclosing funding sources of included trials. Roseman et al have shown that the funding sources of the included trials were fully or partially disclosed in only 46 of 151 Cochrane Reviews and that these disclosures were scattered around seven different places in the reviews.11 The MECIR (Methodological Expectations of Cochrane Intervention Reviews) reporting standards require details of funding sources for each included study and declarations of interest of the primary researchers of the included studies to be placed in the ‘Characteristics of included studies’ table.12 It would be more appropriate to find this information along with other risks of bias in the risk of bias assessment table. Those opposed to adding funding source to the Cochrane risk of bias tool argue that all mechanisms of bias can be identified, perfectly measured, and incorporated quantitatively in the results of meta‐analysis. It is impossible or highly impractical to accurately measure all mechanisms of bias. For example, internal documents from the industry sponsor may be needed to detect how a study was deliberately biased in design or to determine whether outcomes were selectively reported or analysed. It is more realistic to assess bias in studies included in systematic reviews by using empirical methods to identify factors that are associated with research results, as has been done for funding source bias. The impact of the bias can be assessed descriptively or by using subgroup analysis, comparing industry‐funded to non‐industry‐funded studies, as is commonly done in Cochrane Reviews. A bias should not be ignored even if we do not fully understand its mechanism, just as we should not ignore harms of interventions if we don't understand how they arose, or ignore the harm of smoking because we don't know how smoking causes cancer. Therefore, a study's funding source should be evaluated as an independent risk of bias. In summary, the Cochrane risk of bias tool should include funding source as a standard item because: Funding source fits the definition of bias There is empirically‐based evidence of bias related to funding source The observed bias related to funding source cannot be captured by the risk of bias criteria currently assessed with the risk of bias tool Risks of bias are not mutually exclusive Bias may be related to funding source even when all studies are industry‐funded.