Avoiding Catastrophic Forgetting Via Neuronal Decay

In continual learning settings neural network is taught different tasks sequentially and the network is prone to catastrophic forgetting. We investigate the role of regularization methods in this problem. We carry out experiments with permuted MNIST dataset and multilayer perceptron and show that L 1 - and L 2 -regularization on neuronal activation can help to reduce catastrophic forgetting in networks with enough capacity. The proposed neuronal decay significantly outperforms other regularization techniques, including dropout, which is well known in continual learning. The neuronal decay is very simple but powerful, it can be combined with other continual learning methods increasing their peformance. By the use of the proposed technique, we were able to reduce number of errors by up to 20% in gradient projection memory (GPM) method.