How to Estimate Probabilities

By way of introduction, a classification of kinds of probability is given in the form of a tree which also forms an approximate hierarchy: psychological, subjective, logical, physical, and tautological. Various relationships between these kinds of probability are mentioned. Methods, all more or less Bayesian, for the estimation of physical probabilities are then described. Binomial and multinomial probabilities are estimated by means of a three-tiered hierarchical Bayesian method. The method can also be regarded, in some of its aspects, as Bayesian in the ordinary sense, wherein the initial distribution for the physical probabilities is a weighted sum of symmetrical Dirichlet distributions. It can be proved that this is equivalent to the use of a single symmetrical Dirichlet distribution whose parameter is selected after sampling. Thus, in this problem, an ordinary Bayesian method implies an empirical Bayesian method. Next the species-sampling or vocabulary-sampling problem is considered wherein there is a multinomial population having a very large number of categories. The theory derives from a suggestion of Turing's, which in part anticipates the empirical Bayesian method. Among other things, it leads to estimates of population coverage for an enlarged sample. Finally the estimation of probabilities in multidimensional contingency tables is considered. The main method here is that of maximum entropy, but it is shown that this can be subsumed under a more general method of minimum discriminability for the formulation of hypotheses. Entropy is best regarded as a special case of the older and more obviously concept of “expected weight of evidence”.