Information Theory of a Multilayer Neural Network with Discrete Weights

Statistical mechanics is applied to estimate the maximal capacity per weight (α c ) of a two-layer feed-forward network with discrete weights of depth l , functioning as a parity machine of the K hidden units. For each K and l ⩽ l 0 ( K ), the maximal theoretical capacity α c = log 2 (2 l ) is achieved, the capacity per bit is 1, the average overlap between different solutions is zero and l 0 ( K ) log K for large K . At finite temperature, a one-step replica symmetry-breaking solution is found to be exact for l ⩽ l 0 ( K ).