Posterior probabilities for choosing a regression model

SUMMARY A simple Bayesian formula for the posterior probability of one of several regression models is shown to be systematically misleading unless all models have the same number of para- meters. Even in this case the use of improper priors leads to arbitrary inferences, as it does more generally. An alternative weighting for choosing a model is suggested and the relation- ship between significance tests and data-dependent priors mentioned. An alternative to the analysis of variance and related significance tests for choosing regression models is the calculation of the posterior probabilities of the competing models. These two methods can lead to sharply opposed inferences, both asymptotically and for a finite number of observations. It is the purpose of the present paper to examine in detail the potentially misleading behaviour of the apparent posterior probabilities of the models, particularly as a function of the number of observations. In passing, some errors in the literature are identified and some anomalies explained. The calculation of posterior probabilities after each observation arises in a natural way in the analysis of designed experiments for discriminating between regression models. In such experiments sequential procedures are desirable. A formula for updating the posterior probabilities of the models after each observation is given by Box & Hill (1967). In ? 2 the nonsequential form for these posterior probabilities is derived and related to posterior probabilities calculated in the customary way by integration over the parameter space of each model. This relationship makes clear the prior assumptions underlying the sequential procedure. The difficulties which arise from the use of improper priors are also mentioned in ?2. The design of experiments is peripheral to the main discussion of the present paper. But since the numerical investigation of the behaviour of the posterior probabilities requires sequential experiments, design is discussed briefly in ?3. The behaviour of the posterior probabilities is investigated in ?4 for models with the same and with differing numbers of parameters when none, some or all of the models are true. Empirical weights for model plausibilities are derived in ? 5 and related to data-dependent prior probabilities for the models. The argument of the paper is almost entirely conducted in terms of two models with additive independent normal errors of constant known variance. The restriction to two models is purely for convenience, the extension to any number of models being straight- forward as is the extension to linearized nonlinear models. If the variance of the errors is unknown, the expressions for the posterior probabilities of the models become more com- plicated while the properties of the probabilities remain similar to those described here.