Rules and Judgments in Statistics: Three Examples

Statistical analyses are based on a mixture of mathematical theorems and judgments based on subject matter knowledge, intuition, and the goals of the investigator. Review articles and textbooks, aiming for brevity and simplicity, sometimes blur the difference between mathematics and judgment. A folklore can develop, where judgments based on opinions become laws of what "should" be done. This can intimidate authors and readers, waste their time, and sometimes lead to analyses that obscure the information in the data rather then clarify it. Three familiar examples are discussed: the choice between Normal‐based and non‐parametric methods, the use of multiple‐comparison procedures, and the choice of sums of squares for main effects in unbalanced ANOVA. In each case, commonly obeyed rules are shown to be judgments with which it is reasonable to disagree. A greater stress on model selection, aided by informal methods, such as plots, and by informal use of formal methods, such as tests, is advocated.