Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect

As the class size grows, maintaining a balanced dataset across many classes challenging because the data are long-tailed in nature; it is even when the sample-of-interest co-exists with each other in one unit, e. g. , multiple visual instances in one image. Therefore, -tailed classification is the key to deep learning at scale. However, methods are mainly based on re-weighting/re-sampling heuristics that a fundamental theory. In this paper, we establish a causal inference, which not only unravels the whys of previous methods, but also a new principled solution. Specifically, our theory shows that the SGD is essentially a confounder in long-tailed classification. On one, it has a harmful causal effect that misleads the tail prediction biased the head. On the other hand, its induced mediation also benefits the learning and head prediction. Our framework elegantly the paradoxical effects of the momentum, by pursuing the direct effect caused by an input sample. In particular, we use causal in training, and counterfactual reasoning in inference, to remove "bad" while keep the "good". We achieve new state-of-the-arts on three-tailed visual recognition benchmarks: Long-tailed CIFAR-10/-100, -LT for image classification and LVIS for instance segmentation.