Answers are grounded in this paper and cite their sources.
In standard neural network training, the gradients in the backward pass are determined by the forward pass.As a result, the two stages are coupled.This is how most neural networks are trained hitherto.Gradient modification in the backward pass has seldom been studied in the literature.In this paper we explore decoupled training, where we alter the gradients in the backward pass.We propose a simple yet powerful method called PowerGrad Transform (PGT), that alters the gradients before the weight update in the backward pass and significantly enhances the predictive performance of a convolutional neural network.PGT trains networks to arrive at a better optima at convergence.It is computationally efficient, and adds no additional cost to either memory or compute, but results in improved final accuracies on both the training and test datasets.Power-Grad Transform is easy to integrate into existing training routines, requiring just a few lines of code.With decoupled training, our method improves baseline accuracies for ResNet-50 by 0.73%, for SE-ResNet-50 by 0.66% and by more than 1.0% for the non-normalized ResNet-18 network on the ImageNet classification task.
No takes yet. Share an insight, caveat, or question.
Das et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: