A Cyclical Learning Rate Method in Deep Learning Training

The learning rate is an important hyperparameter for training deep neural networks. The traditional learning rate method has the problems of instability of accuracy. Aiming at these problems, we proposed a new learning rate method with different cyclical changes in each training cycle instead of a fixed value. It achieves higher accuracy in less iterations and faster convergence. Through the experiment on CIFAR-10 and CIFAR-100 datasets based on VGG network and RESNET network, the final results show that the proposed method has better results on stability and accuracy than cyclical learning rate method.