How Many Parameters Does the Lasso Use? Stein's Lemma, the Active-Set Degrees of Freedom and an Unbiased Risk Estimate, Tested on a 44-Column Concrete-Strength Design

The lasso both selects and shrinks, so it is not obvious how many parameters it spends. Under Gaussian noise the effective degrees of freedom, defined as the sum of covariances between fitted and observed values divided by the noise variance, equal the expected number of non-zero coefficients (Zou, Hastie and Tibshirani). I give a self-contained proof of Stein's lemma, of the unbiasedness of Stein's unbiased risk estimate (SURE), and of the local divergence formula for the lasso fit, using the fact that the fit is the residual of a projection onto a convex set. I then test the result on a real design: the 1,030 mixes of the UCI Concrete Compressive Strength data with a full second-order expansion of 8 log inputs (44 standardised columns, condition number 6,444) and a semi-synthetic response with 8 true non-zero coefficients and noise sd 7.102 MPa. Over 1,000 replications at six penalty levels, the covariance degrees of freedom and the mean active-set size plus one agree within 0.10, for example 13.92 (se 0.15) against 13.93 at the smallest penalty and 8.41 against 8.38 at the largest. Mean SURE is within one Monte Carlo standard error of the true risk at every penalty. Choosing the penalty by SURE over 200 replications gives a loss 13.0% above the oracle, against 13.6% for 10-fold cross-validation, with one lasso fit per penalty instead of eleven.