What type of study is this?

September 5, 2025Open Access

The Influence of Regularization Intensity on the Bias Variance of Linear Regression

Key Points

The optimal model, with λ = 0.1, achieved a mean squared error of 4.21, outperforming the ordinary least squares model by 15.3%.
Comparative analyses highlighted the ridge model's effectiveness in managing bias-variance trade-off better than traditional OLS.
The methodology involved generating synthetic linear data, implementing gradient descent, and analyzing regularization parameters systematically.
These findings emphasize the importance of regularization techniques, indicating their potential applicability in more complex models like deep neural networks.

Abstract

This study proposes an L2 regularization-based framework for optimizing linear regression models' generalization ability. Through comparative analysis of ordinary least squares (OLS) and ridge-like models on synthetic data, we investigate regularization's role in bias-variance trade-off. The experimental protocol involves: (1) generating linear data (y = 3X + 5 + ϵ) with Gaussian noise (σ = 2), (2) estimating OLS parameters via normal equations, and (3) implementing gradient descent with regularization terms (λ ∈ 0. 0, 0. 01, 0. 1, 1. 0), using 2λθⱼ for weight correction. Results show the λ = 0. 1 model achieves optimal MSE (Mean Squared Error) performance (MSE = 4. 21), 15. 3% better than OLS (MSE = 4. 97), with parameters (intercept = 5. 12, coefficient = 2. 98) closer to true values. Visual analysis confirms the regularized model's superior robustness in feature distribution edges, contrasting with OLS's overfitting tendency. The proposed grid search and gradient correction methods provide an interpretable framework for lightweight model optimization, extendable to elastic networks and deep neural networks in high-dimensional scenarios.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper