Penalized regression with model‐based penalties

Abstract Nonparametric regression techniques such as spline smoothing and local fitting depend implicitly on a parametric model. For instance, the cubic smoothing spline estimate of a regression function ∫ μ based on observations t i, Y i is the minimizer of ΣYi ‐ μ (t i) 2 + λ∫ (μ′′) 2. Since ∫ (μ″) 2 is zero when μ is a line, the cubic smoothing spline estimate favors the parametric model μ (t) = α o + α 1 t. Here the authors consider replacing ∫ (μ″) 2 with the more general expression ∫ (L μ) 2 where L is a linear differential operator with possibly nonconstant coefficients. The resulting estimate of μ performs well, particularly if L μ is small. They present an O (n) algorithm for the computation of μ. This algorithm is applicable to a wide class of L 's. They also suggest a method for the estimation of L. They study their estimates via simulation and apply them to several data sets.