Multiple Comparisons, Explained

The commentary by Savitz and Olshan (1) and the response by Thompson (2) form a fascinating contrast. Savitz and Olshan speak in the language of epidemiology, complaining about the seeming arbitrary and antiscientific dictates of conventional (frequentist) statistical methodology, particularly as it applies to the study of many associations. Thompson, speaking mainly in the language of statistics, takes us on a whirlwind tour of the foundations of statistical inference. He highlights, in particular, the problems with frequentist statistical methodology, supporting in spirit most of the points of Savitz and Olshan, but he ends up condemning the seeming arbitrary and antiscientific consequences of people disobeying the dictates of frequentist statistical methodology. Casual readers could be excused for being confused about what is actually being disputed. Perhaps more relevant, they may not see how the differences claimed here have any real meaning for how epidemiologists analyze, think, or write about their research. In this essay, I will try to show how the issue here is at the core of what epidemiology, and indeed science, is all about. The first step is to identify the real issue for which the methodological arguments are a proxy. The most telling clues are found in the conclusions of each paper. The final sentence of Savitz and Olshan's conclusion states that the context in which ideas originate and the perspective of the investigators offer little guidance in assessing the study's scientific contribution (1, p. 907). Thompson says that if p values are to be used, we need to adjust them to preserve their one merit, the protection against type I errors (2, p. 804). He says the failure to do so, combined with inattention to the method of hypothesis generation, would be license to publish coincidences with a pseudoscientific gloss (2, p. 804).