Questioning Multilevel Models

In this article, practical problems with multilevel techniques are discussed. These problems, brought to our attention by the National Center for Education Statistics (NCES), have to do with terminology, computer programs employing different algorithms, and interpretations of the coefficients in one or two steps. We discuss the usefulness of the hierarchical linear model (HM) in the most common situation in education—that of a large number of relatively small groups. We also point to situations where the more complicated HMs can be replaced with simpler models, with statistical properties that are easier to study. We conclude that more studies need to be done to establish the claimed superiority of restricted versus unrestricted maximum likelihood, to study the effects of shrinkage on the estimators, and to explore the merits of simpler methods such as weighted least squares. Finally, distinctions must be made between choice of model, choice of technique, choice of algorithm, and choice of computer program. While HMs are an elegant conceptualization, they are not always necessary. Traditional techniques perform as well, or better, if there are large groups and small intraclass correlations, and if the researcher is interested only in the fixed-level regression coefficients.