Domains of Solutions and Replica Symmetry Breaking in Multilayer Neural Networks

The relationship between the geometrical structure of weight space and replica symmetry breaking (RSB) in multilayer neural networks is studied using a toy model. The distribution of sizes of the disconnected domains of solution space is computed analytically and compared to the RSB calculation of the Gardner volume. We are able to show explicitly that ergodicity breaking and RSB are not equivalent. Repeating these calculations using the cavity approach allows us to interpret the geometrical meaning of a RSB ansatz. It is well-known [1] that a full study of the storage properties of multilayer neural networks within the Gardner framework [2] requires the breaking of replica symmetry [3]. This differs from the case of the simple perceptron, where a replica symmetric solution has been shown to be exact, and where the spin glass order parameter which emerges has a simple geometrical meaning. In multilayer neural networks, replica symmetry breaking is believed to reflect the existence of a...