Too Many, Too Improbable: Testing joint hypotheses and closed testing shortcuts

This is the authors' abstract. We don't add key points for this paper.

Hypothesis testing is a key part of empirical science and multiple testing as well as the combination of evidence from several tests are continued areas of research. In this article we consider the problem of combining the results of multiple hypothesis tests to (i) test global hypotheses and (ii) make marginal inference while controlling the k -FWER. We propose a new family of combination tests for joint hypotheses, called the ‘Too Many, Too Improbable’ (TMTI) statistics, which we show through simulation to have higher power than other combination tests against many alternatives. Furthermore, we prove that a large family of combination tests – which includes the one we propose but also other combination tests – admits a quadratic shortcut when used in a Closed Testing Procedure, which controls the FWER strongly. We develop an algorithm that is linear in the number of hypotheses for obtaining confidence sets for the number of false hypotheses among a collection of hypotheses and an algorithm that is cubic in the number of hypotheses for controlling the k -FWER for any k greater than one. • We introduce a new family of combination tests. • We show through simulation that our test has high power against many alternatives. • We provide analytical p -value calculations for this family of tests. • We prove a quadratic shortcut for closed testing for a family of combination tests. • We show that m adjusted p -values can be computed in in O ( m 2 ) complexity. • We show that k -FWER control at k > 1 can be obtained in O ( m 3 ) complexity. • We propose an O ( m ) algorithm for finding the number of false hypotheses in a set.