On the probability of making Type I errors.

A statistical test leads to a Type I error whenever it leads to the rejection of a null hypothesis that is in fact true. The probability of making a Type I error can be characterized in the following three ways: the conditional prior probability (the probability of making a Type I error whenever a true null hypothesis is tested), the overall prior probability (the probability of making a Type I error across all experiments), and the conditional posterior probability (the probability of having made a Type I error in situations in which the null hypothesis is rejected). In this article, we show (a) that the alpha level can be equated with the first of these and (b) that it provides an upper bound for the second but (c) that it does not provide an estimate of the third, although it is commonly assumed to do so. We trace the source of this erroneous assumption first to statistical texts used by psychologists, which are generally ambiguous about which of the three interpretations is intended at any point in their discussions of Type I errors and which typically confound the conditional prior and posterior probabilities. Underlying this, however, is a more general fallacy in reasoning about probabilities, and we suggest that this may be the result of erroneous inferences about probabilistic conditional statements. Finally, we consider the possibility of estimating the (posterior) probability of a Type I error in situations in which the null hypothesis is rejected and, hence, the proportion of statistically significant results that may be Type I errors. © 1987 American Psychological Association.