Asymptotic Distribution ofPValues in Composite Null Models

We investigate the compatibility of a null model H 0 with the data by calculating a p value; that is, the probability, under H 0, that a given test statistic T exceeds its observed value. When the null model consists of a single distribution, the p value is readily obtained, and it has a uniform distribution under H 0. On the other hand, when the null model depends on an unknown nuisance parameter θ one must somehow get rid of θ, (e.g., by estimating it) to calculate a p value. Various proposals have been suggested to “remove” θ, each yielding a different candidate p value. But unlike the simple case, these p values typically are not uniformly distributed under the null model. In this article we investigate their asymptotic distribution under H 0. We show that when the asymptotic mean of the test statistic T depends on θ, the posterior predictive p value of Guttman and Rubin, and the plug-in p value are conservative (i.e., their asymptotic distributions are more concentrated around 1/2 than a uni...