The Zeroth Problem

mathematical theory, proceeding from the axiom that rational behavior is coherent, in the sense that it is not rational to accept an incoherent system of bets. But these bets are mathematical idealizations; the real world is more complex than this theory admits. Some Bayesians, for example Lindley (1965), have argued that subjective probabilities belong at the first level, since they can be determined to arbitrary accuracy by elicitation, combined with the requirement of consistency, and so exist objectively (though only inside someone's head). But a corollary of de Finetti's assertion is (pardon me for shouting) SUBJECTIVE PROBABILITY DOES NOT EXIST In my view probability, whether assigned by a frequentist or by a subjective Bayesian, lies firmly at the second level. Phenomena may exhibit stable frequencies and may be unpredictable, but an iid or exchangeable model is only a convenient approximation to reality. Fisher's position in his 1922 paper was very clear. He emphasizes that the probability 1/6 of throwing a five with a die refers to ... a hypothetical population of an infinite number of throws, with the die in its original condition. Our statement will not ... contain any false assumption about the actual die ... or any ... approximation ... Similarly, in my view, personal probabilities are no more than a convenient fiction. I find this view-that a probability is nothing more than a model of reality-to be helpful in resolving the controversy between various schools of thought. Under this view, a frequentist may recognize symmetry among n equally likely cases and so assign a probability 1/n to each. If new evidence suggests that this assignment is inappropriate (perhaps he learns that the die is biased), he may amend this assignment. Similarly, a subjective Bayesian may judge that on his present state of knowledge, some system of bets is acceptable, but if new insights arise he may amend these judgments by means other than Bayes' theorem. (See Leamer (1978), sec. 9.1). In any case, however the probabilities are assigned, the mathematical theory by which theorems are proved belongs at the third level. Thus, in my view, an assignment of probability is like any other scientific theory; not to be believed, but merely for convenience until conflicting evidence is found. This corresponds to the so-called Copenhagen interpretation of quantum mechanics. A difficulty arises-how can we assess the strength of the conflicting evidence? No amount of discrepant evidence can conclusively disprove a probability model. Many discussions remark on the incestuous nature of the situation: the adequacy of the model is to be assessed using the model itself. But there is no difficulty in using the approach suggested above in terms of similarity judgments, with the components of the judgment chosen with a view to the prospective uses of the model. So here I am rejoining the main thread of my argument. 8. THE FEDERALIST AGAIN I would like to expand on this point with respect to the Federalist study. The model MW and we could check explicitly that the frequencies are empirically approximately independent. Assessing the adequacy of this model is a matter of judging whether it is similar (in senses to be determined) to the data. In making this judgment, we might compute various goodness-of-fit measures, but it would be important also to study the sensitivity of the proposed analysis to changes in the model. Thus, MW s3(1O?cr,1O?cr) for T given vr. (MW using data from blocks of known authorship, posterior distributions of the (u, T) parameters (for each word used) could be derived; then using the counts for a disputed paper and integrating out the oand T parameters, an odds ratio for Hamilton versus Madison is obtained. In the event, Madison was strongly favored for each of the 12 disputed papers. The set of words used in this final calculation was chosen after careful study of their discriminatory power in passages of known authorship; but, as M&W remark allowance for selection and regression effects is made entirely through the prior distribuitions of oand -r. We assume that the prior distributions apply to any and every word chosen from some large group of words. Then, according to the model, the prior distribution will reduce the apparent discriminating power to the required extent. With this selection feature, we may choose words for inclusion by whatever methods are convenient, so long as they are independent of the unknown papers. M&W call their whole approach a Bayesian analysis, but this is the only point where a subjective judgment, not based directly on data, is made. What kind of probability is being assumed here? How might such a model be challenged? We would need to find some situation, or preferably several situations, that we judge to be similar to this, and where it is possible to measure somehow whether this assumption is supported by the data. If it is, we can appeal to these analogies to support the assumption in this case. So we would need to consider analyses of other authorship problems (possibly in other languages). A cross-validation study (using subsets of the data and subsets of the words) could contribute to understanding how sensitive the whole analysis is to this assumption.