Effect of initial weights on back-propagation and its variations

The effects is studied on the convergence properties of the back-propagation learning rule of the range from which the initial weight values are randomly selected. In addition to the standard back-propagation rule, two variations are also considered, namely symmetric back-propagation and expected-value back-propagation. In most applications of back-propagation, the range of initial weights is small. It is shown that significantly higher initial weights can substantially improve learning rates. If the initial weight range is increased beyond a problem-dependent limit, however, performance degrades. Symmetric back-propagation is most sensitive to the initial weight range, while expected value back-propagation is least sensitive. The authors describe an improvement on the symmetric variation that produces faster learning rates with low initial weights.>