Assessing random-effects model fit in meta-analysis using a third-order Cochran statistic

In meta-analysis, the conventional random-effects model (REM) is commonly used when heterogeneity is thought probable a priori. However, there are no standard tests of the adequacy of this model. To address this issue, we propose a new third-order Cochran statistic. Our new J index, and associated test, measures departures from the REM, analogous to the way that Cochran’s heterogeneity statistic and the I2 index are used to quantify departures from the fixed effect model. We also introduce a conceptually simple plot that is useful in showing the presence of unusual features of outcome data, that may cast doubt on conventional modelling approaches. A measure of the diversity of study sizes is also introduced. The performance of the proposed methodology is explored in a simulation study and illustrated by analysing some well-travelled datasets. Our proposals can be used as descriptive statistics, to convey the adequacy of conventional statistical methodologies, or help determine model choice.