Random-Effects Meta-Analyses Are Not Always Conservative

It is widely held that random-effects summary effect estimates are more conservative than fixed-effects summaries in epidemiologic meta-analysis. This view is based on the fact that random-effects summaries have higher estimated variances and, consequently, wider confidence intervals than fixed-effects summaries when there is evidence of appreciable heterogeneity among the results from the individual studies. In such instances, however, the random-effects point estimates are not invariably closer to the null value nor are their p values invariably larger than those of fixed-effects summaries. Thus, random-effects summaries are not predictably conservative according to either of these two connotations of the term. The authors give an example from a meta-analysis of water chlorination and cancer in which the random-effects summaries are less conservative in both of these alternative senses and possibly more biased than the fixed-effects summaries. The discussion of when to use random effects and when to use fixed effects in computing summary estimates should be replaced by a discussion of whether summary estimates should be computed at all when the studies are not methodologically comparable, when their results are discernibly heterogeneous, or when there is evidence of publication bias.