Answers are grounded in this paper and cite their sources.
The problem of learning from examples in two-layer neural networks is studied within the framework of statistical mechanics. A fully connected committee machine is trained to implement a task which is not linearly separable. The generalization error as a function of the number of training examples per adjustable weight is calculated in the annealed approximation. For both binary and continuous weights we find a first-order transition with a discontinuous drop in the generalization error. The transitions occur due to a specialization of the hidden units from a symmetric state to one with each hidden unit in the student network specialized on a corresponding unit in the teacher network. The symmetric states of poor generalization remain metastable even for large training sets.
No takes yet. Share an insight, caveat, or question.
Schwarze et al. (1993) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: