An evaluation of homogeneity tests in meta-analyses in pain using simulations of individual patient data

In this paper we consider the validity and power of some commonly used statistics for assessing the degree of homogeneity between trials in a meta-analysis. We show, using simulated individual patient data typical of that occurring in randomized controlled trials in pain, that the most commonly used statistics do not give the expected levels of statistical significance (i.e. the proportion of trials giving a significant result is not equal to the proportion expected due to random chance) when used with truly homogeneous data. In addition, all such statistics are shown to have extremely low power to detect true heterogeneity even when that heterogeneity is very large. Since, in most practical situations, failure to detect heterogeneity does not allow us to say with any helpful degree of certainty that the data is truly homogeneous, we advocate the quantitative combination of results only where the trials contained in a meta-analysis can be shown to be clinically homogeneous. We propose as a definition of clinical homogeneity that all trials have (i) fixed and clearly defined inclusion criteria and (ii) fixed and clearly defined outcomes or outcome measures. In pain relief, for example, the first of these would be satisfied by all patients having moderate or severe pain, whilst the second would be satisfied by using at least 50% pain relief as the successful outcome measure.