Comprehensive Survey of Distributed Optimization: Challenges, Strategies, and Applications

Distributed optimization plays an important role in multi-node collaborative computing and has developed rapidly in recent years, both in theory and applications. However, most classical distributed optimization algorithms are generally developed under idealized communication settings, which, to some extent, limit their applicability to practical systems with complex constraints. This survey aims to bridge the gap between theory and practice in distributed optimization by examining its theoretical foundations, recent progress, and major open challenges. Initially, we outline some theoretical breakthroughs regarding graph topologies, objective functions, and acceleration strategies. The discussion then turns to approaches for addressing practical constraints like stochastic noise, limited bandwidth, and asynchronous delays. Finally, we present several directions that are worth exploring in the future.