A memory constrained bayesian optimization via robust online memory estimation

Bayesian optimization (BO) is a memory-intensive algorithm that requires training and evaluating an expensive objective function. In contrast to previous works that use an offline memory estimation to make BO memory-efficient, we propose a robust and simple online memory estimation method that requires training a model only for the first two iterations of the first epoch. Our memory estimation method is then integrated with a simple, performance-based surrogate model of BO in a seamless (or in sync) mode that enforces memory efficiency even if it does not bypass a preset threshold. The online memory estimation method has been evaluated on two different datasets, showing that it is more accurate than the existing offline method ( 2.19× 2.19 × for MNIST and 3.51× 3.51 × for CIFAR datasets). Furthermore, compared to a memory-unaware baseline, the enhanced BO has no loss of accuracy and is 11.31× 11.31 × memory-efficient for a simple CNN-based image classification, and 5.03× 5.03 × memory efficient but 9.27× 9.27 × slower for a more complex LSTM-based text classification (useful for a resource-constrained environment where delay is tolerable but memory is scarce), while it is 2.6× 2.6 × memory efficient but 1.23× 1.23 × slower on a pretrained ResNet50 model.