A Comparison of GCV and GML for Choosing the Smoothing Parameter in the Generalized Spline Smoothing Problem

The partially improper prior behind the smoothing spline model is used to obtain a generalization of the maximum likelihood (GML) estimate for the smoothing parameter. Then this estimate is compared with the generalized cross validation (GCV) estimate both analytically and by Monte Carlo methods. The comparison is based on a predictive mean square error criteria. It is shown that if the true, unknown function being estimated is smooth in a sense to be defined then the GML estimate undersmooths relative to the GCV estimate and the predictive mean square error using the GML estimate goes to zero at a slower rate than the mean square error using the GCV estimate. If the true function is "rough" then the GCV and GML estimates have asymptotically similar behavior. A Monte Carlo experiment was designed to see if the asymptotic results in the smooth case were evident in small sample sizes. Mixed results were obtained for n = 32, GCV was somewhat better than GML for n = 64, and GCV was decidedly superior for n = 128. In the n = 32 case GCV was better for smaller ² and the comparison close for larger ². The theoretical results are shown to extend to the generalized spline smoothing model, which includes the estimate of functions given noisy values of various integrals of them.