In random effects meta-analysis with a small number of studies and considerable between-study variation, the actual significance level of the standard hypothesis test can be much larger than nominal.
The random effects model is routinely used in meta-analysis. Although a variety of approaches have been proposed for implementing this in practice, the most popular method is that suggested by DerSimonian and Laird. The resulting inferences are not exact, however, and modifications to the methodology have been suggested. By deriving the exact distribution of the resulting test statistic, under the null hypothesis and in the simplified scenario where all studies are the same size, it is possible to obtain the actual significance level of the standard hypothesis test for a treatment effect. In particular it is found that, if there is a small number of studies and considerable between-study variation, the actual significance level can be much larger than the nominal level. This and other findings are illustrated using examples concerning the effectiveness of treatments to reduce serum cholesterol for preventing death in patients with no history of heart attacks, the use of glycerol for patients who have suffered an acute stroke, and the use of amisulpride to treat schizophrenia.
Dan Jackson (2009) studied this question. Random effects model (DerSimonian and Laird) was evaluated on Actual significance level of the standard hypothesis test. In random effects meta-analysis with a small number of studies and considerable between-study variation, the actual significance level of the standard hypothesis test can be much larger than nominal.