What question did this study set out to answer?

The aim is to explore the performance of prediction models and the estimation of prediction error.

synapse

⌘+K

synapse

⌘+K

August 24, 2004

The Estimation of Prediction Error

Key Points

The aim is to explore the performance of prediction models and the estimation of prediction error.
Constructed data-based estimation rules using logistic regression and classification trees.
Examined penalty methods like Akaike's information criterion and cross-validation techniques.
Derived a connection between nonparametric methods and penalty counterparts through a Rao-Blackwell relation.
Model-based penalty methods demonstrated significantly better accuracy when the model assumptions held true.
Cross-validation methods are shown to be randomized versions of covariance penalty methods.

Abstract

Having constructed a data-based estimation rule, perhaps a logistic regression or a classification tree, the statistician would like to know its performance as a predictor of future cases. There are two main theories concerning prediction error: (1) penalty methods such as Cp, Akaike's information criterion, and Stein's unbiased risk estimate that depend on the covariance between data points and their corresponding predictions; and (2) cross-validation and related nonparametric bootstrap techniques. This article concerns the connection between the two theories. A Rao–Blackwell type of relation is derived in which nonparametric methods such as cross-validation are seen to be randomized versions of their covariance penalty counterparts. The model-based penalty methods offer substantially better accuracy, assuming that the model is believable.

Mark Helpful

Bookmark

Relay