On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

The stochastic gradient descent (SGD) method and its variants are algorithms choice for many Deep Learning tasks. These methods operate in a small-batch wherein a fraction of the training data, say 32-512 data points, is to compute an approximation to the gradient. It has been observed in that when using a larger batch there is a degradation in the quality the model, as measured by its ability to generalize. We investigate the for this generalization drop in the large-batch regime and present evidence that supports the view that large-batch methods tend to to sharp minimizers of the training and testing functions - and as is known, sharp minima lead to poorer generalization. In contrast, -batch methods consistently converge to flat minimizers, and our support a commonly held view that this is due to the inherent noise the gradient estimation. We discuss several strategies to attempt to help-batch methods eliminate this generalization gap.