Simultaneous Test Procedures--Some Theory of Multiple Comparisons

When a hypothesis is tested by a significance test and is not rejected, it is generally agreed that all hypotheses implied by that hypothesis (its "components") must also be considered as non-rejected. However, when the hypothesis is rejected the question remains as to which components may also be rejected. Various writers have given attention to this question and have proposed a variety of multiple comparisons methods based either on tests of each one of the components or on simultaneous confidence bounds on parametric functions related to the various hypotheses. An approach to such methods, apparently originally due to Tukey 27, is to test each component hypothesis by comparing its statistic with the level critical value of the statistic for the overall hypothesis. This is called a Simultaneous Test Procedure (STP for short) as all hypotheses may be tested simultaneously and without reference to one another. An STP involves no stepwise testing of the kind employed by some other methods of multiple comparisons for means, in which subsets are tested for equality only if they are contained in sets which have already been found significant. (See 3, 4, 10, 18). A general formalization of STP's is attempted in this paper. Section 2 introduces the requisite concepts of families of hypotheses and the implication relations between them, as well as the monotonicity relations between the related statistics. Section 3 defines STP's and shows conditions for coherence and consonance of their decisions, these properties being that hypothesis implication relations are preserved in the decisions of the STP. Section 4 discusses comparison of various STP's for the same hypotheses and shows the advantages of the union-intersection type of statistics and of reducing the family of hypotheses tested as much as possible. Section 5 translates all these results to simultaneous confidence statements after introducing the definitions necessary to allow such translation. The analogy between simultaneous test and confidence methods is of special importance as it brings a wide spectrum of methods within this framework, most of which was originally formulated in confidence region terms. This covers the original work of Tukey 27 and Scheffe 25 and continues with that of Roy and his associates 21 and most recently Krishnaiah 12, 13. A general discussion of this confidence approach has been given by Aitchison 1 since the first draft of the present paper. In view of the close analogies pointed out in Section 5, it is not surprising that Aitchison arrives at the requirement of a constant critical value for all his tests, exactly as in an STP. In fact his approach and the present one are complementary. The formal theory is illustrated with ANOVA examples to clarify the concepts. No essentially new techniques are presented in this paper though this approach has been used elsewhere by the author to derive a number of practically useful procedures 6, 7, 8.