A Stochastic Recursive Gradient Algorithm with Inertial Extrapolation for Non-convex Problems and machine learning

In recent years, extrapolation acceleration technology has been widely employed across various algorithms; however, its application in machine learning has yielded limited achievements. Therefore, building upon the foundation of inertial acceleration technology, we conducted a study and analysis of the small-batch variant of the Stochastic Average Recursive Adaptive Hyperparameter (SARAH) algorithm. By integrating extrapolation acceleration technology into the SARAH framework, we developed an inertial stochastic recursive gradient method tailored for addressing empirical risk minimization problems in scenarios involving non-convex loss functions and machine learning. In this paper, under specific moderate assumptions, we have demonstrated the convergence of the algorithm to the global optimum. Furthermore, we have derived its linear convergence rate with respect to the non-convex objective function. The numerical results obtained from evaluating two widely utilized machine learning problems clearly illustrate the superior performance and practical feasibility of the proposed algorithm.