Speeding Up Hyperparameter Optimization of Deep Neural Networks: A Review of Multi-Fidelity-Based Methods

Deep Learning models are usually long to train and evaluate that performing an adequate hyperparameter optimization is intractable. Different algorithms based on fidelity evaluations, also known as multi-fidelity algorithms, have been proposed to speed up the hyperparameter optimization process so far. This paper aims to review the state-of-the-art algorithms in multi-fidelity and define an accurate taxonomy to classify them. It includes an accurate description of the strengths and weaknesses of these algorithms not only from a descriptive but also an experimental perspective. These experiments involved different problems and conditions to perform a fair evaluation of the proposal’s performance and computational cost.