Bayesian Cluster-Adjusted Standardised Mean Differences: A Simulation Study

Reporting standardised mean differences of continuous psychometric measures is omnipresent in psychology. In a randomised controlled trial with a multilevel structure, one can define a cluster-adjusted version of these quantities naturally taking into account heterogeneous residual variances and accounting for the multilevel structure. In this article, we carefully define the underlying location-scale distributional mixed-effects models and derive cluster-adjusted standardised mean differences using partial pooling. Estimating these quantities is best handled using Bayesian statistics, and we perform an extensive Monte Carlo simulation study to test the performance of these metrics under different simulation configurations. We conclude that in order to measure the a-priori heterogeneous treatment effect across clusters in practically relevant scenarios, it is enough to consider random effects only on the location model and not on the scale model. Even under artificial severe heteroscedasticity, this reduced model performs well both in terms of parameter bias and predictive accuracy. The resulting cluster-adjusted standardised mean difference is an average value across clusters, where the heterogeneity is summarised via its standard deviation.