Statistical control versus classification of study quality in meta-analysis

A vexing problem for meta-analysts is how to account for the differential methodological quality of studies. We critically examine five approaches that deal with methodological quality. The first approach is to ignore the methodological quality of studies and aggregate all available research for which effect sizes (ES) can be calculated. The second approach is to use methodological quality as an inclusion criterion, thereby excluding all studies that fail to meet some predetermined level of quality. The third approach is to use methodological quality to classify included studies and report their findings separately. The fourth approach is to weight studies by methodological quality. And the fifth approach is to treat methodological quality as a predictor of ES by removing its influence from the collection of evidence. After discussing some of the limitations of the five approaches, we describe a method for treating methodological quality as a study feature, coding for it, and adjusting for its influence. A four-step procedure is illustrated using two examples. Step one: Are the ES homogeneous? Step two: does study quality explain the heterogeneity? Step three: which qualities of studies matter? Step four: how do we deal with the differences?