On-line backpropagation in two-layered neural networks

We present an exact analysis of learning a rule by on--line gradient descent in a two--layered neural network with adjustable hidden--to--output weights (backpropagation of error). Results are compared with the training of networks having the same architecture but fixed weights in the second layer. PACS.: 07.05.Mh, 87.10, 02.50 The ability of neural networks to learn a rule from examples 1 has been studied successfully in a statistical mechanics context, see e.g. 2, 3, 4 for recent reviews. So far most of the analysis has been restricted to very simple networks like the single layer perceptron 1 or networks with one layer of hidden units and a fixed hidden--to--output relation, e.g. the so--called committee machine 3. In the following we extend the recent investigation of learning by on--line gradient descent 8, 10 to two--layered networks with adjustable weights connecting the hidden units and the output. This topic is of crucial importance as systems with variable hidden--...