Weighted UnLearning from Experience for Improved Spurious Correlation Robustness

Despite their success, Deep Learning (DL) models are susceptible to spurious correlations. These are accidental correlations between non-predictive features of the training dataset and the target label, leading to models that do not generalise well to samples without such spurious correlations. We propose to address this failure mode by extending and improving an existing method for spurious correlation robustness — UnLearning from Experience (ULE). ULE trains two models in parallel; a biased model is encouraged to learn the spurious correlations, while a separate debiased model is trained to solve the same task while avoiding learning the same behaviour as the first, thus avoiding learning the spurious correlations. A weakness of ULE is that it treats every training example identically, regardless of whether or not it contains a spurious correlation. Therefore, ULE does not make optimal use of the training data, leading to worse-than-necessary performance. To address this weakness, we introduce Weighted UnLearning from Experience (wULE), which modifies the original ULE loss function to reweight the components depending on whether a training sample likely contains spurious correlations. Hence, we explore two methods for identifying training samples likely to contain spurious correlations. We test our approach using several well-known spurious correlation robustness benchmarks — Waterbirds, CelebA, UrbanCars and MultiNLI. Our novel wULE method outperforms the original ULE in all cases and achieves Worst-Group Accuracy (WGA) results comparable to the best methods in the spurious correlation robustness literature.