AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

Most popular optimizers for deep learning can be broadly categorized as methods (e. g. Adam) and accelerated schemes (e. g. stochastic gradient (SGD) with momentum). For many models such as convolutional neural (CNNs), adaptive methods typically converge faster but generalize compared to SGD; for complex settings such as generative adversarial (GANs), adaptive methods are typically the default because of their. We propose AdaBelief to simultaneously achieve three goals: fast as in adaptive methods, good generalization as in SGD, and training. The intuition for AdaBelief is to adapt the stepsize according to "belief" in the current gradient direction. Viewing the exponential moving (EMA) of the noisy gradient as the prediction of the gradient at the time step, if the observed gradient greatly deviates from the prediction, distrust the current observation and take a small step; if the observed is close to the prediction, we trust it and take a large step. We AdaBelief in extensive experiments, showing that it outperforms other with fast convergence and high accuracy on image classification and modeling. Specifically, on ImageNet, AdaBelief achieves comparable to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief high stability and improves the quality of generated samples to a well-tuned Adam optimizer. Code is available at: //github. com/juntang-zhuang/Adabelief-Optimizer