Converting between effect sizes

This Campbell methods policy note on converting between effect sizes sets out current Campbell Collaboration policy on calculating a common effect size from different metrics for use in meta-analyses in Campbell systematic reviews of intervention effects. It is intended to serve as a point of reference for Campbell authors and editors in both the design and production phases of Campbell reviews of intervention effects, to ensure that protocols and full reviews are prepared in line with collaboration policy. In order to provide context for the current policy, the Note is prefaced with a brief introduction to the topic and appended with a list of additional resources, tables of formulas and worked examples. It is not intended to provide comprehensive guidance or a tutorial on how to apply methods in Campbell or other systematic reviews (see below). Campbell authors and editors should consult Campbell Collaboration systematic reviews: Policies and Guidelines for detailed information on the general requirements for Campbell systematic reviews, guidelines for producing them, and selected sources of further information about systematic reviews that are consistent with those requirements and guidelines. Campbell authors and editors should also refer to Methodological expectations of Campbell reviews of intervention effects (MEC2IR) – standards for the conduct and reporting of Campbell Collaboration systematic reviews of intervention effects. MEC2IR conduct and reporting standards can be found at https: //campbellcollaboration. org/library. These standards provide authors and users of Campbell reviews of intervention effects with clear and transparent expectations of review conduct and reporting and facilitate the editorial process. All new or updated Campbell reviews of intervention effects that proceed to the full review production stage (i. e. following approval of the protocol) after 1 October 2014 are required to comply with all mandatory MEC2IR conduct and reporting standards in order to be signed-off for publication in the Campbell systematic reviews monograph series. MEC2IR conduct standards #47 (‘making maximal use of data’), #62 (‘combining different scales’) and #63 (‘ensuring meta-analyses are meaningful') address issues related to converting between effect sizes. Campbell also publishes Campbell Methods Guides that provide detailed guidance for authors on how to apply specific methods in the production of Campbell systematic reviews. Campbell Methods Guides supplement general guidelines for producing Campbell systematic reviews, as described in Campbell Collaboration systematic reviews: policies and guidelines. Campbell Methods Guides do not currently cover the topic of converting between effect sizes. Statistical meta-analysis is the preferred method for integrating studies in a systematic review of intervention effects (Campbell Collaboration, 2014). However studies identified for inclusion in a systematic review typically report a range of different measures of the same outcome construct. Without a common effect size metric across studies, the analyst has limited options to synthesize studies and compare findings across contexts. Therefore, before researcher can use statistical meta-analysis to synthesize findings from different studies estimates of effects from individual studies must be converted into a common metric. The purpose of this Methods Policy Note is to provide an introduction to transforming effect sizes into common metrics in order to synthesize studies and to outline the Campbell policy in this area. An effect size is a measure of the magnitude of an observed relationship, treatment effect, or population parameter (Kelley and Preacher, 2012). Studies included in systematic reviews of effects typically provide effect sizes which estimate the magnitude of the difference in an outcome between participants in the treatment groups and participants in the control group. An effect size is different from tests of statistical significance, which indicate whether results are likely to be due to chance. Tests of statistical significance tell us nothing about the practical significance of the results in terms of the size of the effect (Ellis, 2010). To answer this question, we need an effect size. There are many different types of effect size indices, including standardized mean-difference (d), odds ratio (OR), and correlation coefficient (r). In the case of the standardized mean-difference, an analyst divides the difference between two group's means by the standard deviation. A standard deviation represents the spread of the data, assuming that the distribution is reasonably large and normal. Using this principle, we are able to quantify the mean-difference in relation to the spread from which it derives. For example, an intervention designed to increase reading scores may result in the comparison group scoring a sample mean of 50 and a sample standard deviation of 10, and the intervention group scoring a sample mean of 80 and a sample standard deviation of 10. The principle of the standard deviation states that 68. 2% of all participants’ scores will fall within one standard deviation above and below the mean. Therefore, 68. 2% of score will fall between 40 – 60 in the comparison group and 68. 2% of scores will fall between 90 – 110 in the intervention group. Figure 1 illustrates this concept. Distribution of variable scores for intervention and control groups. A standardized mean-difference can be calculated using the two distributions' means and standard deviations The odds ratio and correlation coefficient also represent the magnitude of an effect, but these are calculated differently because of the nature of the measures. An odds ratio can measure the size of the effect when the outcome variable is dichotomous. A correlation coefficient can also measure the effects of a treatment, but instead of measuring the difference between the two groups' means, the correlation coefficient observes the relationship between the treatment status and the outcome. Each of these concepts are related, but the results of the different effect size calculations are not directly comparable. This is because the effects are in different “metrics”, or the scale of the effect size. This is easy to visualize in the Table 1. The standardized mean-difference and correlation coefficient each have “no effect” centered at 0. 0, but the odds ratio's center is at 1. 0. The standardized mean-difference has no theoretical lower and upper limits, while the correlation coefficient is standardized to fall between -1. 0 and 1. 0. The odds ratio has a lower limit of 0. 0 with no theoretical upper limit. To conduct a meta-analysis across all of the effect sizes, the analyst must transform all effect sizes into a common metric. The metric that all effect sizes should be transformed to depends on the measures and data provided by the included studies. A synthesis containing ten studies and ten effects may have seven standardized mean-difference effect sizes and three odds ratios. The analyst should transform the smallest number of effect sizes possible, in this case, transforming the three odds ratios into standardized mean-difference effect sizes. An example of this can be found in the Campbell review on school-based programs to decrease teen dating violence (De La Rue, Polanin, Espelage, Borenstein et al, 2009; Lipsey and standardized mean difference to correlation coefficient) using the program R (R Core Team, 2016). Additionally, there are many adjustments and modifications that may be made for the conversion from odds ratio to a correlation coefficient. The conversion presented in Appendix A assumes that no additional information about the marginal is provided in the primary study. Should a 2 times 2 table be provided, Bonnett b) perform sensitivity analyses, especially when using educated guesses or imputation to supply information for the transformation, and c) conduct moderator analyses that test for differences between un-transformed and transformed effect sizes (see Lipsey 75 individuals are included in each group. d <-. 35 a <- ( (75 + 75) ²) / (75*75) rd <- d/sqrt ( ( (d²) +a) ) rd ## 1 0. 1724 Vd <- ( (75 + 75) / (75 * 75) ) + ( (d²) / (2* (75 + 75) ) ) Vᵣd <- ( (a²) *Vd) / ( ( (d²) +a) ²) Vᵣd ## 1 0. 02549