Reproducible research and the substantive context

In a recent editorial in Biostatistics, Peng (2009) introduced a system for making data and computer code from papers in the journal publicly available so that other researchers can reproduce the statistical analyses in the paper. The editors explain in a short accompanying note (Diggle and Zeger, 2009) that ‘we wish to encourage the practice of making research published in the journal reproducible by others’. Peng's proposal follows an earlier commentary by Peng and others (2006) that had caught the attention of the influential journal Annals of Internal Medicine (cf. Laine and others, 2007). I agree that it is commendable to be able to reproduce the statistical calculations that a group of authors has done, and Peng has put much care into developing a framework that should allow precise documentation of data and computer code. Hopefully, the focus on reproducibility will enhance the level of documentation of original research data and of the actual calculations that were done, a documentation which may sometimes leave quite a bit to be desired. Peng's editorial also actively encourages others to conduct “alternative analyses.” This could sometimes be constructive if the purpose was to develop new methodology, but then the term “reproducible” is a misnomer. From the context, it seems that the purpose of these alternative analyses concerns the substantive issues, and my intention with this note is to point out that this is less innocent than it may look at first glance. We have severe caricatures in our tradition that should make us careful. There is usually much more to “research” than the statistical analysis, and there is much more to the statistical analysis than the correctness of calculations. A reasonable choice of statistical model (here “correct” is not a meaningful descriptor) requires deep insight into the substantive context, but it is quite unclear how a satisfactory role for the substantive context is assured in the new system. We should avoid encouraging the already far too widespread occurrence of poorly informed reanalyses of old data sets about which the statistician knows almost nothing, a culture that apparently bothers editors of (other!) statistical journals surprisingly little. In my view, there is no guarantee that the “reproduction” that comes out of this culture will benefit science. Much of today's biostatistical research is deductive or “method-driven,” often in the form that some existing method is generalized. There is a fairly standard template for this type of research: explain the generalization, give an (often superficial) justification that asymptotics will probably work, run a simulation study to see that this is credible, and conclude with a brief and often explicitly disinterested outline of a so-called “real data example,” where the substantive importance of the questions that the new method can solve is rarely an issue, neither for the author nor, surprisingly, for the editor. The real data example all too often comes from another paper in a statistical journal, and there can easily be many generations of publications since the substantive scientist had a say in what is being done to his or her data or the involved statistician knew what the substantive problem really was. Preece (1986) gave a sarcastic discussion with several examples of this practice, particularly from textbooks. It does happen that such method-driven research brings our field forward, but in my view real progress is more often made when the research is inductive or “problem-driven.” That is, a collaboration between substantive scientists and statisticians has identified a substantive problem that requires new statistical methodology for its satisfactory solution. The statistician will develop the method, by which new substantive evidence is revealed, and, in the best of cases, new directions for statistical research are also identified. Such new directions are not necessarily so mathematically convenient but will on the other hand often open new paths for methodological research. And: the substantive problems involved are important and well-informed! Journal editors should give preference to this type of biostatistics. Reporting of problem-driven biostatistics will almost always involve collaboration between statistician and substantive scientist since here we are involved with important real problems that have often necessitated many resources in acquiring the data, and both parties have to make sure that everything is well understood in the final publication from the methodological as well as the substantive point of view. The Journal of the American Medical Association requires that clinical trials sponsored by drug companies have to be reanalyzed by “academic statisticians” before acceptance for publication in the journal (Journal of the American Medical Association, 2004, Fontanarosa and others, 2005). This requirement has generated widespread debate among biostatisticians, eloquently summarized by Imrey (2007). I sat on panels discussing the issue at the International Biometric Conference in Montreal in 2006 and the Joint Statistical Meetings in Salt Lake City in 2007. One objection from the biostatisticians is the general mistrust that this policy displays against a specific segment (those working for industry) of our colleagues. I will here focus on another objection: the actual mechanical SAS- (or R-) number crunching of the finally developed master data set is a minor part of the totality of the statistical analysis, and it ridicules our profession to believe that there is a serious check on reproducibility in seeing if somebody else's computer reaches the same conclusion using the same code on the same data set as the original statistician's computer did. During this debate, it was repeatedly hammered out that what ivory tower-statisticians may sometimes believe is an objective unfallible basic set of data is in fact a result of countless concrete judgments made in the course of the study primarily by the substantive scientists but usually with insightful input from the study statisticians. This is where an important part of the statistician's responsibility lies, but it is impractical to devise a reanalysis template that can capture this whole process. To draw substantive conclusions from a reanalysis without awareness of the prehistory is unprofessional and trivializes biostatistics. On the outlined background I was curious to see how Biostatistics would create a constructive approach to reproducibility. It turns out that the substantive context has almost been ignored. Peng's implicit assumption is that there exists some immutable data set, directly understandable to any statistician, whose judgments about the relevant substantive context are apparently believed to be directly relevant and credible. The only vague hint to the possibility that the data may require insight is the possibility in Peng's item 3 to submit “External data or auxiliary files containing … other required information” as if such information would only rarely be needed. Otherwise, the focus is on the technical tools for the numbers and the code. If reanalysis is to make sense more broadly than checking the calculations, there at least has to be sufficient information to make it realistic for another interdisciplinary group of researchers to understand the substantive context and the strengths and weaknesses of the data. Another possibility might in principle be that the original substantive collaborators make themselves available as partners in reanalyses, and some might put this as a condition for releasing the data. My guess is, however, that this desirable option would rarely be chosen. The journal should rethink how due respect be paid to the complex contexts that our work is involved with in most nontrivial situations. Otherwise, the quest for reproducibility may lead us further into the inbreeding where statisticians lecture each other about the conclusions of data sets for which the real context has long been forgotten, and our colleagues in the substantive sciences will not be impressed.