Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-Tailed Noise and Power of Symmetry

Key Points

• Improved mean squared error (MSE) rates are observed under heavy-tailed noise conditions, enhancing performance significantly.
• In the analysis, large deviation upper bounds are derived for nonlinear stochastic gradient descent (SGD).
• Observational analysis focusing on the interplay between heavy-tailed noise and the power of symmetry provides new insights.
• Highlights the importance of understanding noise characteristics in optimizing nonlinear SGD applications.