P Values as Random Variables—Expected P Values

Abstract p values are extensively reported in practical hypothesis testing situations. Although carefully studied by Dempster and Schatzoff, the stochastic aspect of p values is often neglected. In this expository note we borrow from Dempster and Schatzoff to rekindle interest in—and explore the potential usefulness of—understanding the stochastic behavior of p values. We relate the expected p value (EPV) under the alternative to the more familiar concepts of significance level and power. We then go on to argue that in cases where it is difficult to evaluate the power function, the EPV can be used as a measure of the performance of a test. EPV's are always easily evaluated or simulated. Different test statistics for the same hypotheses can also be compared by means of EPV's. We carry out such a comparison between the two-sample, one-sided Kolmogorov-Smirnov, Mann-Whitney, and t tests, for a variety of underlying distributions. The EPV can also be a valuable tool in sample size determination and in the interpretation of observed p values. We hope to convince practitioners of the usefulness of EPV's. Key Words: Comparison of testsExpected p valuePowerRandom p value