On Constructing Confidence Intervals for a Standardized Mean Difference in Meta-analysis

Under the random effects model for meta-analysis, confidence intervals for the overall effect are typically constructed using quantiles of the standard normal distribution. We discuss confidence intervals based on both the standard normal distribution and the t-distribution, in conjunction with several methods of estimating the heterogeneity variance for a standardized mean difference, and we compare the empirical coverage probabilities of the intervals using simulation. The coverage probabilities of intervals based on an approximate t-statistic are higher than the coverage probabilities for the standard normal intervals, and are very close to the specified confidence level even for small meta-analysis sample size. Moreover, intervals based on the approximate t-statistic appear relatively robust to different methods of estimating the heterogeneity variance, unlike the normal intervals. Thus, we conclude that confidence intervals based on the t-statistic are superior to the standard normal confidence intervals for a standardized mean difference, and should be used by practitioners in place of the normal intervals.