Regularisation and the least squares problem in the analysis of echo state networks

An echo state network (ESN) is a recurrent neural network that has several advantages with respect to a deep neural network, including its fast training phase and the absence of vanishing and exploding gradients. The training phase reduces to solving the least squares (LS) problem w ls = arg min v || vX - y || 2 , where X is the reservoir matrix, and X and y are functions of the training data. It is common to add regularisation to this problem because, it is claimed, it minimises the adverse effects of overfitting. Recent work in deep neural networks, physics informed neural networks and regression has shown, however, that regularisation does not solve the problem of overfitting, and thus this paper considers the application of regularisation to ESNs by analysing their predictive abilities on several time series, including a non-linear communication channel, the Hénon map and multiple superimposed oscillations. It is shown that the solution w ls of the LS problem is stable with respect to a perturbation in y for a wide range of parameter values of an ESN, and thus regularisation must not be applied. In particular, regularisation is not benign because its use when a condition on the rate of decay of the singular values of X is not satisfied leads to a large error between the theoretically exact and regularised solutions of the LS problem.