A learning rule eliminating local minima in multilayer perceptrons

Convergence problems in the case of the generalized delta rule are discussed. On the basis of the analysis performed, a modification to the nonlinearity of processing elements is proposed; this modification is shown to smooth the cost function to be minimized during the learning phase. A variation to the generalized delta rule learning procedure, required by the introduced modification, is discussed. Extensive tests have been performed on several examples proposed in the technical literature. The tests show the effectiveness of the proposed procedure in improving the convergence properties of the backpropagation algorithm. In particular, it has been verified that the proposed modification virtually eliminates nonconvergence problems of a moderate eta value is used.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>