On-line learning in the committee machine

The dynamics of learning from examples in the K=3 non-overlapping committee machine with single presentation of examples is studied. The optimal algorithm, in the sense of mean generalization, is obtained from a variational analysis of the differential equations which describe the dynamics. The agreement of the theoretical predictions and the results of numerical simulations is excellent. The optimized dynamics has the extra advantage with respect to the non-optimized cases in that it uncouples the differential equations which describe the evolution of the relevant parameters, i.e. the student-teacher overlap and the norm of the student synaptic vector. This, in turn, translates into the possibility of constructing useful practical optimized on-line algorithms. For the optimal algorithm the generalization error decays as approximately 0.88 alpha -1, the same nominal error as for the simple perceptron with optimized dynamics.