Editorial: What to do with Missing Data?

In the Pearsonian–Fisherian era, with the advent of proper experimentation with biological systems, came the headaches accompanying incomplete data. What worried our early day colleagues most was destruction of balance or other design features. Although, for example, an analysis of variance with unequally sized groups is possible, it implies a bigger challenge for the computer than its conventional, equally sized group counterpart. Remember that, in Fisher's time, the computer was, instead of our current day laptop, a room full of assistants. In those days, they typically were women, since Fisher was of the for us now unimaginable opinion that women should do the calculations the men did not want to do themselves. Although the difference between 1 and 7 μs is inconsequential, it used to be, for example, 7 days for an unbalanced analysis of variance rather than ‘only’ 1 day for the balanced one. Then, with a focus shift from biological and agricultural experimentation to clinical trials and observational studies in humans, supplemented with sociological, psychometric and econometric research, the prevalence of incompleteness rose dramatically around and after World War II. The dominant concern to restore balance and to maintain design simplicity has naturally led to such methods as complete-case (CC) analysis and simple imputation methods. In other words, balance was restored by either taking bits out or by filling in gaps. It is fair to say that such methods were centrefield for roughly half a century. In line with an unfortunate tradition in applied statistics, these methods come under a variety of names: what is called CC by the biometrician may be listwise deletion in sociology and beyond, much to the confusion, not only of our substantive colleagues, but unfortunately also of ourselves. Although some alternative methods had been developed, in an attempt to avoid both deletion and artificial creation of data, including such imputation methods as available case analysis or even ad hoc instances of the expectation–maximization (or EM) algorithm, it is fair to say that the advent of missing data methodology as a genuine field within statistics, with its proper terminology, taxonomy, notation and body of results, was initiated by Rubin's (1976) landmark paper. Rubin brought together scattered pieces from the literature and fitted these within his own pivotal methodological work. As a result, we now routinely distinguish between monotone and non-monotone missingness, missingness completely at random (MCAR), missingness at random (MAR) and missingness not at random (MNAR) mechanisms, ignorable and non-ignorable inferences, selection, pattern–mixture, and shared parameter models, etc. Together with other work, prominently the formalization of the EM algorithm ( Dempster et al., 1977) and multiple imputation (MI) ( Rubin, 1987), these fundamental concepts are nicely laid out in Little and Rubin (1987), the first reference ever on missing data; this or the second edition ( Little and Rubin, 2002) should be on every self-respecting statistical practitioner's shelf. However, it is naïve to assume that a paradigm shift took place in the late 1980s or early 1990s. Users happily stuck to CC and imputation methods, in particular the last observation carried forward (LOCF) method, i.e. using the last observed value as a substitute for all subsequent missing measurements), within a regulated clinical trial environment. Reasons for this are manifold. First, there is the usual inertia, owing in part to the complexity of, for example, the drug development process, and the existence of guidelines, rules and regulations. The LOCF method has become somewhat of a fetish in certain circles. Although some regulatory agencies are moving beyond this method, along with other simplistic ones, unfortunately not all seem to follow suit. Second, spreading taxonomy and concepts across users from a wide range of backgrounds and with varying levels of methodological aptitude, though desirable as such, unavoidably generates haze or even plain confusion as a side-effect. The recent applied literature abounds with conceptual confusion. For example, one happily conducts simulations to compare CC and pattern–mixture models and an ignorable data analysis; this makes little sense, since these three concepts belong to three different taxonomical dimensions. First, CC analysis belongs to the class of MCAR, along the mechanism dimension. Second, pattern–mixture models are one of the families along the model framework dimension. Along the third dimension, ignorable and non-ignorable data analyses are contrasted. Getting the concepts and their implications right in one's mind is of the essence. Third, even though good methods are available, they are of little practical value when they are not accompanied by proper computational devices. Although the times of human computers were well behind us during the 20th century's closing decade, and plenty of good statistical software was available indeed, this was not so for what should already then have been standard missing data methods. The emergence of software for longitudinal data analysis made the handling of highly unbalanced data, whether or not this imbalance is provoked by missingness, routine business from, say, the mid-1990s. Even more recently, decent MI implementations have entered the community. As a fourth and final issue slowing down a paradigm shift, the field may perceive that there is a blatant lack of synthesis among theorists. Let us expand on this crucial current day issue. In 1994, Robins, Rotnitzky and Zhao wrote a seminal paper on the semiparametric analysis of incomplete data ( Robins et al., 1994). It sparked off an important fresh strand of research, with implications for different but related fields such as causal inference. van der Laan and Robins (2003) synthesizes this work. As the Rothamsted legends have it and, more tangibly, older discussions of Royal Statistical Society meeting papers bear witness to, sharp controversy among statisticians is perennial. The ‘R2’ issue, between the Rubin and Robins schools, is but another one of these. There among researchers in the missing data field widely known controversies are multilayered, including, for example, the choice between likelihood or Bayesian methods on the one hand and frequentist, semiparametric techniques on the other hand. Controversy is to some extent beneficial, as it prevents us from lazily choosing the comfort of well-trodden paths, but it also risks preventing a synthesis from emerging. Instead of digging trenches, researchers should advance methodological development that would combine the best of both schools, such as MI combined with generalized estimating equations, rather than purely likelihood-based MI, from the Rubin school, or pure weighted generalized estimating equations, from the Robins side. Syntheses like these can emerge once we acknowledge that all methods have advantages and disadvantages; this is like the usual trade-off between bias and efficiency, which incidentally is to some extent the choice that the user is facing in the choice between parametric and semiparametric equipment. The Robins school has brought forward the beautiful theory of double robustness, giving maximal protection against model misspecification. It is accessibly described by Tsiatis (2006). The last decade has witnessed an explosion of research in the MNAR category, where missingness can depend on unobserved outcomes, in addition to information that is already offered by the observed outcomes. It is a tricky situation, since we are considering models where explanatory power is extracted from data … which we do not have. Now this is not uncommon, e.g. the prevalence of random effects, latent classes, latent groups and the like. However, such models invariably are haunted by unverifiable assumptions. This is fine when not going unacknowledged and, after all, it ought not to be forgotten that the unverifiable assumptions area is already involved when assuming that the missing data process is missingness at random (MAR). Arguably, such models should not be used for primary analysis, but preferably they find their place within a so-called sensitivity analysis. We could define this as an analysis in which several statistical models are considered simultaneously and/or where a statistical model is further scrutinized by using specialized tools, such as diagnostic measures to detect influential observations or subjects. Sensitivity tools are actively being developed and a very preliminary synthesis can be found in Molenberghs and Kenward (2007), but we should and undoubtedly will see much more work in this area. Hopefully, methodologists will think of pairing up their creations with software tools. In conclusion, it is time to place CC analysis and simple imputation methods, in particular LOCF, in the Museum of Statistical Science, and to consider instead ignorable likelihood analysis, ignorable Bayesian analysis, MI and weighted estimating equations as the standard. In other words, one should enter the building at the MAR floor and not at the MCAR level. This is an assignment for statisticians in academe, industry and Government, in particular those working for regulatory agencies around the globe. More work is needed at the MNAR level; again, not so much in search of the ultimate model that definitively overcomes the missingness problem, because such a model is statistics's philosopher's stone. More realistically, emphasis should be on furthering the subfield of sensitivity analysis. We do not want to look for the ultimate model but rather we need to know what models do to our conclusions, given the availability of only a portion of the designed data. Additionally, the integration of substantive information into our modelling or sensitivity analysis exercise should be taken much more seriously than is currently the case. Learning our lesson from history, it is our duty to accompany new methodology with reliable, flexible, software implementations; this is a duty of the individual researcher and the software houses alike.