A New Generalization of Gradient Method with Penalty and Momentum Terms for Wavelet Feedforward Neural Networks

Regularization plans, acceleration mechanisms, and optimization strategies all have a significant impact on the convergence behavior of neural network training algorithms. A thorough analysis of a batch gradient method (BGM) improved with momentum and penalty terms for training wavelet feedforward neural networks (WFNNs) is presented in this research. In nonlinear function approximation challenges, the suggested approach seeks to enhance convergence speed, stability, and generalization performance. To prove the monotonicity, weak convergence, and strong convergence of the training process under the combined influence of momentum regularization and penalty, a rigorous mathematical framework is created. While the momentum term speeds up convergence by reducing oscillations in the gradient trajectory, the penalty term suppresses weight overgrowth and overfitting. The algorithm is applied to the Peaks function, Sinc function, and 5-bit parity problem to support these theoretical findings. The findings show that in terms of quicker convergence, lower mean squared error (MSE), and smaller gradient norms, the suggested BGM with penalty and momentum performs better than traditional BGM with L2 regularization. Smoother optimization dynamics and improved generalization are revealed by the training convergence and gradient norm evolution curves. The suggested method offers a cohesive viewpoint on integrating momentum acceleration, regularization, and penalty into gradient-based optimization for WFNNs. It is a promising technique for function approximation, pattern recognition, and control system applications since it provides both theoretical convergence guarantees and real-world increases in learning efficiency.