Alternative Methods for Assessing Mediation in Multilevel Data: The Advantages of Multilevel SEM

Abstract Multilevel modeling (MLM) is a popular way of assessing mediation effects with clustered data. Two important limitations of this approach have been identified in prior research and a theoretical rationale has been provided for why multilevel structural equation modeling (MSEM) should be preferred. However, to date, no empirical evidence of MSEM's advantages relative to MLM approaches for multilevel mediation analysis has been provided. Nor has it been demonstrated that MSEM performs adequately for mediation analysis in an absolute sense. This study addresses these gaps and finds that the MSEM method outperforms 2 MLM-based techniques in 2-level models in terms of bias and confidence interval coverage while displaying adequate efficiency, convergence rates, and power under a variety of conditions. Simulation results support prior theoretical work regarding the advantages of MSEM over MLM for mediation in clustered data. Keywords: mediationmultilevel modelingmultilevel SEMstructural equation modeling Notes 1For example, 1–2–2 denotes a design in which the independent variable X is assessed at Leve1 1, whereas both the mediator M and outcome Y are assessed at Level 2. 2ICC is often interpreted as the proportion of variability in a variable that is between-cluster. 3 CitationHox (2002) noted that school research often reports ICCs of .10 to .15, whereas small group and family research often reports ICCs in the neighborhood of .15 to .30. CitationSnijders and Bosker (1999) and B. O. Muthén (1991, 1994) indicated that ICCs of .05 to .20 are common. CitationJulian (2001) used ICCs of .05, .15, and .45 in his simulation study. We consider values of .05 small, .10 medium, and .20 large. 4We did not examine unbalanced cluster sizes, as Lüdtke et al. (2008) found no effect of varying cluster size in 1–1 models. We considered manipulating J and nj such that the total sample size would remain constant (CitationJulian, 2001; Z. Zhang et al., 2009) but instead chose to separately manipulate J and nj to examine the separate and joint effects of changing the number of clusters and the number of cases sampled within cluster (Lüdtke et al., 2008). 5The CIs we used are based on the multivariate delta method and incorrectly assume the indirect effect to be normally distributed, and therefore symmetric about the point estimate. Because indirect effects typically are not normally distributed in small samples, we expect coverage to be a little lower than .95 even under the best of circumstances. Whereas this kind of CI suffices for comparing methods in a simulation, in practice we recommend using a different kind of CI that does not assume the indirect effect to be symmetrically distributed (see Discussion). 6Simulation results support the predictions of Equation 10 for the UMM method. Consistent with Equation 10, in Table 1 we see that RPB does not seem to depend on J. For comparison purposes, Equation 10 predicts RPB values of −57.00, −45.00, −30.00, and −15.00 for the four ICC conditions when n j = 5, −49.57, −25.71, −12.00, and −4.62 when n j = 20, and −39.31, −13.85, −5.45, and −1.94 when n j = 50. 7In an unreported single-level analysis condition, all 2,000 runs converged in every cell. However, single-level analysis is highly inappropriate for hierarchical data. On the other hand, when ICC is low, the Between covariance matrix can become singular or near-singular. Lüdtke et al. (2008) encountered more frequent estimation errors in low-ICC conditions of their simulation. CitationJulian (2001) found that the disadvantages of using single-level approaches rather than multilevel approaches when ICC < .05 were minimal. In this case, rescaled χ2 statistics and robust SE estimation can be used to correct the minimal degree of bias (B. O. Muthén & Satorra, 1995). In Mplus version 5.21, convergence rates are dramatically improved with respect to earlier versions of Mplus for all methods, but particularly for MSEM at low nj and low ICC. 8See http://www.quantpsy.org/