Problems in the medical interpretation of overviews

Recently, investigators have combined formally the results of all available randomized trials testing a particular therapy to get a better estimate of the effectiveness of that treatment than any single trial can provide in isolation. It seems intuitively clear, however, that formal overviews will yield medically meaningful results only under certain defined circumstances. First, the treatments that are pooled should be similar enough that any inferences about the effect of treatment will refer to something more specific than an idealized 'average therapy'. Second, patient selection in the pooled trials should be uniform enough that the inferences will be applicable to some defined patient population. Third, although the potential biases of excluding trials from pooling are substantial, so are the problems of including trials whose execution might be flawed in a biased manner; it is not clear what can be done about this, since biased studies probably cannot be identified in an unambiguous manner. Finally, it would seem prudent to consider the medical context during which trials were performed; it is probably not reasonable to assume that quantitative measures of treatment effect obtained by pooling studies from different eras of treatment will be an accurate reflection of what current treatment can achieve, even if everything else is held constant. For these reasons the quantitative measures of treatment effect that derive from formal overviews may have little relevance to medical decision-making. Overviews might still be useful in indicating the general direction of a treatment effect, provided that no qualitative interactions are present. Although such interactions may seem improbable, some recent examples from the cancer literature suggest that their presence cannot be ruled out a priori.