Prediction Via Orthogonalized Model Mixing

We introduce an approach and algorithms for model mixing in large prediction problems with correlated predictors. We focus on the choice of predictors in linear models, and mix over possible subsets of candidate predictors. Our approach is based on expressing the space of models in terms of an orthogonalization of the design matrix. Advantages are both statistical and computational. Statistically, orthogonalization often leads to a reduction in the number of competing models by eliminating correlations. Computationally, large model spaces cannot be enumerated; recent approaches are based on sampling models with high posterior probability via Markov chains. Based on orthogonalization of the space of candidate predictors, we can approximate the posterior probabilities of models by products of predictor-specific terms. This leads to an importance sampling function for sampling directly from the joint distribution over the model space, without resorting to Markov chains. Compared to the latter, orthogonalized model mixing by importance sampling is faster in sampling models and is also more efficient in finding models that contribute significantly to the prediction. Further advantages are in the speed of convergence and the availability of more reliable convergence diagnostic tools. We illustrate these in practice, using a data set on prediction of crime rates. The model space is small enough so that enumeration of all models is available for comparison and convergence checks. Also, we demonstrate the feasibility of orthogonalized model mixing in a large-size problem, which is very difficult to attack by other methods. The data set is from a designed experiment dealing with predicting protein activity under different storage conditions. The model space is large (the rank of the design matrix is 88) and very difficult to explore if expressed in terms of the original variables. We obtain prediction intervals and a probability distribution of the setting that produces the highest response.