A New Hyperparameter Tuning Framework for Regression Tasks in Deep Neural Network: Combined-Sampling Algorithm to Search the Optimized Hyperparameters

This paper introduces a novel hyperparameter optimization framework for regression tasks called the Combined-Sampling Algorithm to Search the Optimized Hyperparameters (CASOH). Our approach enables hyperparameter tuning for deep learning models with two hidden layers and multiple types of hyperparameters, enhancing the model’s capacity to work with complex optimization problems. The primary goal is to improve hyperparameter tuning performance in deep learning models compared to conventional methods such as Bayesian Optimization and Random Search. Furthermore, CASOH is evaluated alongside the state-of-the-art hyperparameter reinforcement learning (Hyp-RL) framework to ensure a comprehensive assessment. The CASOH framework integrates the Metropolis-Hastings algorithm with a uniform random sampling approach, increasing the likelihood of identifying promising hyperparameter configurations. Specifically, we developed a correlation between the objective function and samples, allowing subsequent samples to be strongly correlated with the current sample by applying an acceptance probability in our sampling algorithm. The effectiveness of our proposed method was examined using regression datasets such as Boston Housing, Critical heat flux (CHF), Concrete compressive strength, Combined Cycle Power Plant, Gas Turbine CO, and NOx Emission, as well as an ‘in-house’ dataset of lattice-physics parameters generated from a Monte Carlo code for nuclear fuel assembly simulation. One of the primary goals of this study is to construct an optimized deep-learning model capable of accurately predicting lattice-physics parameters for future applications of machine learning in nuclear reactor analysis. Our results indicate that this framework achieves competitive accuracy compared to conventional random search and Bayesian optimization methods. The most significant enhancement was observed in the lattice-physics dataset, achieving a 56.6% improvement in prediction accuracy, compared to improvements of 53.2% by Hyp-RL, 44.9% by Bayesian optimization, and 38.8% by random search relative to the nominal prediction. While the results are promising, further empirical validation across a broader range of datasets would be helpful to better assess the framework’s suitability for optimizing hyperparameters in complex problems involving high-dimensional parameters, highly non-linear systems, and multi-objective optimization tasks.