Approximation of Least Squares Regression on Nested Subspaces

For a regression model yᵢ = θ(xᵢ) + εᵢ, the unknown function θ is estimated by least squares on a subspace Λₘ = span\ψ₁, ψ, ⋯, ψₘ\, where the basis functions ψᵢ are predetermined and m is varied. Assuming that the design is suitably approximated by an asymptotic design measure, a general method is presented for approximating the bias and variance in a scale of Hilbertian norms natural to the problem. The general theory is illustrated with two examples: truncated Fourier series regression and polynomial regression. For these examples, we give rates of convergence of derivative estimates in (weighted) L₂ norms and establish consistency in supremum norm.