Funnels for publication bias – have we lost the plot?

With an average of 75 trials published in peer-review journals every day 1, it can get pretty complicated even for a full-time, professional data analyst to keep up with the flood of information 2. Meta-analysis, using a set of statistical methods to integrate data from independent studies, is a god-send, as much of the groundwork of searching for relevant studies, assessing their reliability, risk of bias, and summarising the results, has already been done and results are presented in bite-size chunks in the form of ‘forest-plots’ 3. But before we rub our hands in glee at the thought of all the time we have saved trawling through data, let us give some thought to an often used method to detect reporting bias in meta-analyses, the funnel plot. A funnel plot is a scatter diagram used to visually depict the relationship between the effect of an intervention/treatment (x axis) and study precision (y axis), of the study (Fig. 1) 4. From a statistical point of view, the precision of the estimated intervention effect will increase with increase in study size (presuming the event rate was the same across all studies) 5, and the dots representing such larger studies will appear at the top of the funnel (a in Fig. 1), while in contrast, effect estimates from smaller studies may be expected to scatter near the bottom of the funnel (b in Fig. 1). ‘Small-study effects’ is a term used to describe a trend for smaller studies in a meta-analysis to show larger treatment effects 6; one reason for this may be publication bias, where the chances of a small study being published is higher if the study shows a ‘statistically significant’ effect 7. Although it should be noted that the term ‘small-study effect’ for this phenomenon is somewhat misleading, since large studies can be imprecise, just as small studies can be precise. We propose that ‘imprecise-study effects’ may be a better description. In the absence of publication bias, if both large and smaller (and both precise and imprecise) studies are published, the scatter plot should resemble a funnel. In the presence of publication bias, there will be asymmetry, as though a bite has been taken out of the funnel (Fig. 2) 8. In standard texts about publication bias, the missing chunks always appear at the bottom, on the supposition that it is easier to file-drawer smaller studies, rather than large studies involving multi-centres, though this has recently been shown to be untrue 9. And though putting studies in the file drawer may contribute to funnel plot asymmetry, it certainly is not the only cause of the appearance of large bites out of the funnel. The paper describing the use of funnel plot asymmetry for test of bias in meta-analyses by Egger 8 is one of the most highly cited articles in the history of the British Medical Journal, with over 10 000 citations to date (July 2015, Web of Science). Despite Egger's careful explanation of the possible causes of funnel plot asymmetry, asymmetry is more often than not solely attributed to publication bias. But in addition to publication bias and pure chance, what other factors contribute to funnel plot asymmetry? True heterogeneity in intervention effects is one such factor. For example, a significant effect may only be seen in high-risk patients, and these patients were more likely to be recruited into the smaller, early trials. Larger, multi-centre interventions involving hundreds of patients may be more difficult to implement when compared with smaller trials, and, in addition, there may be methodological differences between centres. In such a case, the data from the smaller, better-controlled study may be more precise than the larger, and perhaps less vigorously implemented study involving a more heterogeneous group of participants. Funnel plot shapes can also be influenced by the statistic used to measure the effect size 8, 10. Figure 3a–c are funnel plots showing the same data of total postoperative morphine consumption in two groups after pre-emptive treatment. While using the statistic standard difference in means (Fig. 3a) and correlation (given as Fisher's z, Fig. 3b) will result in a similar looking funnel plot, when the difference in means is used (without standardisation, Fig. 3c), a completely different shaped funnel is seen. The standardised mean difference is a popular statistic employed when the studies in a meta-analysis measure the same outcome, but may have used different scales, for example, the primary outcome could be acute postoperative pain, but the scales used range from visual analogue scale for pain (VAS) 0 to 10, VAS 0 to 100, or numerical rating scale. The ‘standardised mean difference’ is calculated by dividing the difference in mean outcome between groups by the standard deviation of outcome among participants. This means that we can use the standardised mean difference as an effect size, no matter what scales were used to collect the data 11. In addition to standardised mean difference, ‘Hedges g’ and ‘Cohen's d’ 12 may also be used for continuous variables, and you will often come across these effect size measures in meta-analyses. While categorical and time-to-event data has to be dealt with using ‘hazard ratios’, ‘risk ratios’, ‘proportional odds ratios’ among many others, all of which will impact the shape of the funnel. Even if all meta-analysts could come to a consensus on the statistic used for expressing effect size for each type of variable, there still remains a concern that visual interpretation of symmetry is essentially subjective. In fact, surveys to test the ability of researchers to identify ‘asymmetry’ in funnel plots found that asymmetry was only correctly detected around half the time, that is, no better than chance 3. Statistical tests for asymmetry can be applied, but used only when there are more than ten studies. For studies using standardised mean differences, the test for asymmetry proposed by Egger 8 can be used. Egger's asymmetry test employs a linear regression approach, with the intercept of the regression line providing a measure of asymmetry. For dichotomous outcomes, tests proposed by Peters 13, which are similar to linear regression analyses, plotting odds ratios against the inverse of the total sample size, can be used. Though statistical methods exist to detect funnel plot asymmetry, we must remember that funnel plots themselves can only answer this question, that is, do studies with little precision (studies with few participants) give different results from studies with greater precision (studies with more participants). No matter how well we can detect asymmetry, we cannot deduce from the plot how many studies have been conducted and not published due to non-favourable results. True measures of publication bias would require all investigators to adhere to pre-registration on trial registers and for meta-analysts to compare published studies with those unpublished, but documented in the registries 14. Though meta-analyses may seem to be able to relieve the busy clinician of the arduous task of having to read dozens of individual articles, we must still exercise caution when interpreting their conclusions. No external funding and no conflicts of interest declared.