Prediction Error and Its Estimation for Subset-Selected Models

Strategies are compared for development of a linear regression model and the subsequent assessment of its predictive ability. Simulations were performed as a designed experiment over a range of data structures. Approaches using a forward selection of variables resulted in slightly smaller prediction errors and less biased estimators of predictive accuracy than all possible subsets selection but often did not improve on the full model. Random and balanced data splitting resulted in increased prediction errors and estimators with large mean squared error. To maximize predictive accuracy while retaining a reliable estimate of that accuracy, it is recommended that the entire sample usually he used for model development and assessment.