Finite-size effects in on-line learning of multilayer neural networks

PACS. 87.10+e – General, theoretical, and mathematical biophysics (including logic of biosystems, quantum biology, and relevant aspects of thermodynamics, information theory, cybernetics, and bionics). PACS. 02.50−r – Probability theory, stochastic processes, and statistics. Abstract. – We complement recent advances in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multilayer networks by calculating fluctuations possessed by finite-dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate specific teacher vectors, increasing with the degree of symmetry of the initial conditions. In light of this, we include a term to stimulate asymmetry in the learning process, which typically also leads to a significant decrease in training time. An attractive feature of neural networks is their ability to learn a parametrised rule from a set of input/output training examples, by which the parameters of the network are adapted to minimise an error measuring the misfit of the network mapping on the training examples. Different approaches to the learning process are typically evaluated by the expected error