Robust Loss Functions for Learning Multi-class Classifiers

Robust learning in presence of label noise is an important problem of current interest. Training data often has label noise due to subjective biases of experts, crowd-sourced labelling or other automatic labelling processes. Recently, some sufficient conditions on a loss function are proposed so that risk minimization under such loss functions is provably tolerant to label noise. The standard loss functions such as cross-entropy or mean-squared error, used for learning neural network classifiers, do not satisfy these conditions. It was shown that a loss function based on mean absolute value of error satisfies the conditions and is also empirically seen to be robust to label noise. However, minimizing absolute value of error is a difficult optimization problem. In this paper we propose a new loss function, called robust log loss and show that it satisfies the sufficient conditions for robustness. The resulting optimization problem of minimizing empirical risk is well behaved. Through extensive empirical results we show that, in terms of accuracy and learning rate, the proposed loss function is as good as cross-entropy loss for learning neural network classifiers when there is no label noise and that it is better when the training data has label noise.