Performance of Between-study Heterogeneity Measures in the Cochrane Library

The growth in comparative effectiveness research and evidence-based medicine has increased attention to systematic reviews and meta-analyses. Meta-analysis synthesizes and contrasts evidence from multiple independent studies to improve statistical efficiency and reduce bias. Assessing heterogeneity is critical for performing a meta-analysis and interpreting results. As a widely used heterogeneity measure, the I 2 statistic quantifies the proportion of total variation across studies that is caused by real differences in effect size. The presence of outlying studies can seriously exaggerate the I 2 statistic. Two alternative heterogeneity measures, the JOURNAL/epide/04.03/00001648-201811000-00010/inline-graphic1/v/2023-09-08T093746Z/r/image-tiff and JOURNAL/epide/04.03/00001648-201811000-00010/inline-graphic2/v/2023-09-08T093746Z/r/image-tiff have been recently proposed to reduce the impact of outlying studies. To evaluate these measures’ performance empirically, we applied them to 20,599 meta-analyses in the Cochrane Library. We found that the JOURNAL/epide/04.03/00001648-201811000-00010/inline-graphic3/v/2023-09-08T093746Z/r/image-tiff and JOURNAL/epide/04.03/00001648-201811000-00010/inline-graphic4/v/2023-09-08T093746Z/r/image-tiff have strong agreement with the I 2 , while they are more robust than the I 2 when outlying studies appear.