Answers are grounded in this paper and cite their sources.
Stochastic optimization methods have been hugely successful in making large-scale optimization problems feasible when computing the full gradient is computationally prohibitive. Using the theory of modified equations for numerical integrators, we propose a class of stochastic differential equations that approximate the dynamics of general stochastic optimization methods more closely than the original gradient flow. Analyzing a modified stochastic differential equation can reveal qualitative insights about the associated optimization method. Here, we study mean-square stability of the modified equation in the case of stochastic coordinate descent.
No takes yet. Share an insight, caveat, or question.
Giovacchino et al. (2024) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: