Structural hyperparameter selection (HPS) in neural network (NN) regression faces two critical, computationally expensive barriers: the mandatory splitting of datasets for validation, which significantly impairs sample efficiency, and the inability of conventional metrics (like Data MSE) to decouple true modeling error from detrimental output noise, leading to suboptimal architectural complexity and overfitting. To resolve these systemic limitations, we propose the Reconstruction Error Minimization for Hyperparameter Selection (REM-HPS) framework, a novel, non-Bayesian approach grounded in statistical learning theory. REM-HPS fundamentally shifts the optimization objective by minimizing the Reconstruction Mean Squared Error (MSE), which precisely isolates and measures the model’s intrinsic ability to recover the underlying noise-free function. Since this target error is typically inaccessible, the framework employs the observable Data MSE (validation error) to construct a reliable, probabilistic estimate, yielding a deterministic and noise-aware selection criterion. REM-HPS utilizes a deterministic structural hyperparameter selection criterion that removes randomness due to validation data splitting, while remaining compatible with standard stochastic training procedures. This strategy allows for the use of the entire dataset for training, eliminating the need for explicit data splitting or the introduction of tuning-intensive regularization hyperparameters. Rigorous empirical validation demonstrates that REM-HPS consistently selects significantly more compact architectures (minimal complexity) while achieving superior generalizability and estimation accuracy, particularly across varied Signal-to-Noise Ratios and data regimes. By providing an efficient and optimal selection metric, REM-HPS offers a transformative, resource-efficient alternative to structural HPS in modern data-driven systems.
Beheshti et al. (Sun,) studied this question.