Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

Due to the high computational demands executing a rigorous comparison between optimization (HPO) methods is often cumbersome. The goal of this is to facilitate a better empirical evaluation of HPO methods by benchmarks that are cheap to evaluate, but still represent realistic cases. We believe these benchmarks provide an easy and efficient way to reproducible experiments for neural hyperparameter search. Our consist of a large grid of configurations of a feed forward neural on four different regression datasets including architectural and hyperparameters concerning the training pipeline. Based on data, we performed an in-depth analysis to gain a better understanding of properties of the optimization problem, as well as of the importance of types of hyperparameters. Second, we exhaustively compared various state-of-the-art methods from the hyperparameter optimization on these benchmarks in terms of performance and robustness.