Search for promising points of neural networks initialization

The paper examines the problem of initialization, which consists in determining the starting point in the space of the neural network weights for a qualitative gradient descent. It is known that the learning speed and learning result of a neural network mostly depends on its initialization. The problem is that in the process of execution, the algorithm can get into the anomalous zone of the local minimum and get stuck there for a long time, sometimes at the very beginning of training. In such cases, it is recommended to start the descent with a new initialization. The study proposes to start training from a point at the bottom of the local minimum, from which, as research shows, the descent in a new direction begins. The proposed method is based on the assumption that most of these anomalies are critical points that have a direction to descent. The neural network is initialized every time the learning algorithm encounters an anomaly in which the value of the error function increases. At the same time, a new initialization is determined at the minimum point of the error function in the segment connecting the point before the anomaly and the following one. Successive initializations result in the descent occurring along an optimized trajectory without significant oscillations. The effectiveness of the method is demonstrated by the example of a perceptron with a single hidden layer and one anomaly. It is proved that the descent from the new initialization point is faster; this demonstrates the possibility of gradient correction without calculating the Hessian; a conclusion is made about the expediency of multiple initializations of the neural network when it enters the zone of the next anomaly. It is proposed to use the method of correcting the gradient descent algorithm to reach the bottom of the anomaly in cases where the error begins to grow. The developed method gives a high probability of hitting a region with a low value of the error function than in the case of a random drawing of the initialization point, as well as a benefit in comparison with a Restricted Boltzmann’s machine in terms of execution time.