Investigation of Ways to Handle Sampling Weights for Multilevel Model Analyses

When analysts estimate a multilevel model using survey data, they often use weighted procedures using multilevel sampling weights to correct the effect of unequal probabilities of selection. This study addresses the impacts of including sampling weights and the consequences of ignoring them by assessing the performance of four approaches: the multilevel pseudo–maximum likelihood (MPML), the probability-weighted iterative generalized least squares (PWIGLS), the naive (ignoring sampling weights), and the sample distribution methods for a linear random-intercept model under a two-stage clustering sampling design. When inclusion probabilities are correlated with the values of outcome variable conditioning on the model covariates, the sampling design becomes informative. The results show that whether a sampling design is informative and at which stage of the sampling design it is informative have substantial impacts on the estimation. The results also show that the level of variation of sampling weights is correlated with the bias of estimates. A higher level of variation of sampling weights is associated with a higher level of bias when a sampling design is informative; however, under a noninformative design, the level of variation of sampling weights may not necessarily associate with biased results. Ignoring an informative sampling design at the first stage will result in biased estimates on the intercept and variance of random effect, whereas ignoring an informative sampling design at the second stage will lead to slightly underestimated fixed effects and residual variance, in addition to the biased estimates on the intercept and variance of random effect. Including the sampling weights as the hybrid methods (MPML and PWIGLS) may still produce biased estimates on the intercept and variance of random effect and slightly underestimated fixed effects and residual variance. The sample distribution method may give unbiased estimates, but it depends on the correct specification of the sampling process.