Generalized p -Values in Significance Testing of Hypotheses in the Presence of Nuisance Parameters

Abstract This article examines some problems of significance testing for one-sided hypotheses of the form H 0: θ ≤ θ 0 versus H 1: θ > θ 0, where θ is the parameter of interest. In the usual setting, let x be the observed data and let T (X) be a test statistic such that the family of distributions of T (X) is stochastically increasing in θ. Define Cx as X: T (X) — T (x) ≥ 0. The p value is p (x) = sup θ≤θ0 Pr (X ∈ Cx | θ). In the presence of a nuisance parameter η, there may not exist a nontrivial Cx with a p value independent of η. We consider tests based on generalized extreme regions of the form Cx (θ, η) = X: T (X; x, θ, η) ≥ T (x; x, θ, η), and conditions on T (X; x, θ, η) are given such that the p value p (x) = sup θ≤θ0 Pr (X ∈ Cx (θ, η) ) is free of the nuisance parameter η, where T is stochastically increasing in θ. We provide a solution to the problem of testing hypotheses about the differences in means of two independent exponential distributions, a problem for which the fixed-level testing approach has not produced a nontrivial solution except in a special case. We also provide an exact solution to the Behrens—Fisher problem. The p value for the Behrens—Fisher problem turns out to be numerically (but not logically) the same as Jeffreys's Bayesian solution and the Behrens—Fisher fiducial solution. Our approach of testing on the basis of p values is especially useful in multiparameter problems where nontrivial tests with a fixed level of significance are difficult or impossible to obtain.