Robust Standard Errors and Confidence Intervals for Standardized Mean Difference

The Student's t-test assumes common population variance for testing raw score mean difference (RSMD). The Cohen-Glass-Hedges standardized mean difference (SMD) align with this assumption and are primarily applicable to experimental designs where an intervention affect mean but not variance. However, the intervention may increase or decrease variance due to differential responses driven by individual differences. Furthermore, variances may differ between natural groups in observational data, a phenomenon that is well-documented in various fields of research. One prevalent suggestion is to use the Welch-Satterthwaite t-test for RSMD. Nonetheless, SMD counterparts has been lagging behind. Various methods has been offered in the literature but they often either fail to preserve the definitions of Cohen-Glass-Hedges SMD or are inconsistent with ordinary least square (OLS) framework. This study proposes a consistent framework between RSMD tests and SMDs using OLS sandwich estimators (one of which is precisely same as the Welch-Satterthwaite t-test). For each proposed RSMD testing procedure, we derive a consistent SMD counterpart using the delta method and construct confidence intervals that are based on lambda-prime distribution. Compared to conventional Cohen-Glass-Hedges SMD, these new indices offer efficiency gains, allowing direct application in hypothesis testing of point null, non-inferiority, superiority, equivalence, and minimal effect testing even without raw data. They also enable more precise power calculations. Finally, they improve efficiency of between-study heterogeneity indices in meta-analysis. Implications are discussed and some guidelines are provided.