Weight Space Structure and Internal Representations: A Direct Approach to Learning and Generalization in Multilayer Neural Networks

We analytically derive the geometrical structure of the weight space in multilayer neural networks in terms of the volumes of couplings associated with the internal representations of the training set. In this framework, focusing on the parity and committee machines, we show how to deduce learning and generalization capabilities, both reinterpreting some known properties and finding new exact results. The relationship between our approach and information theory as well as the Mitchison-Durbin calculation is established. Our results are exact in the limit of a large number K of hidden units, whereas for finite K a complete geometrical interpretation of symmetry breaking is given.