Linear and Nonlinear Extension of the Pseudo-Inverse Solution for Learning Boolean Functions

We consider in this letter the pseudo-inverse solution for the learning of a binary classification. We address the problem of overfitting, i.e. the fact that the generalization rate can be relatively low although the learning rate is very high. We interpret this phenomenon with respect to the behaviour of the small eigenvalues of the covariance matrix of the learned patterns. We propose two ways for solving this problem: the first one is linear, the second one is a two-layer perceptron. Numerical simulations are given to illustrate these approaches.