On-line backpropagation in two-layered neural networks

We present an exact analysis of learning a rule by on--line gradient descent in a two--layered neural network with adjustable hidden--to--output weights (backpropagation of error). Results are compared with the training of networks having the same architecture but fixed weights in the second layer. PACS.: 07.05.Mh, 87.10, 02.50 The ability of neural networks to learn a rule from examples [1] has been studied successfully in a statistical mechanics context, see e.g. [2, 3, 4] for recent reviews. So far most of the analysis has been restricted to very simple networks like the single layer perceptron [1] or networks with one layer of hidden units and a fixed hidden--to--output relation, e.g. the so--called committee machine [3]. In the following we extend the recent investigation of learning by on--line gradient descent [8, 10] to two--layered networks with adjustable weights connecting the hidden units and the output. This topic is of crucial importance as systems with variable hidden--...