Asymptotic Distribution of P Values in Composite Null Models

Abstract We investigate the compatibility of a null model H 0 with the data by calculating a p value; that is, the probability, under H 0, that a given test statistic T exceeds its observed value. When the null model consists of a single distribution, the p value is readily obtained, and it has a uniform distribution under H 0. On the other hand, when the null model depends on an unknown nuisance parameter θ one must somehow get rid of θ, (e.g., by estimating it) to calculate a p value. Various proposals have been suggested to “remove” θ, each yielding a different candidate p value. But unlike the simple case, these p values typically are not uniformly distributed under the null model. In this article we investigate their asymptotic distribution under H 0. We show that when the asymptotic mean of the test statistic T depends on θ, the posterior predictive p value of Guttman and Rubin, and the plug-in p value are conservative (i.e., their asymptotic distributions are more concentrated around 1/2 than a uniform), with the posterior predictive p value being the more conservative. In contrast, the partial posterior predictive and conditional predictive p values of Bayarri and Berger are asymptotically uniform. Furthermore, we show that the discrepancy p value of Meng and Gelman and colleagues can be conservative, even when the discrepancy measure has mean 0 under the null model. We also describe ways to modify the conservative p values to make their distributions asymptotically uniform.