Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks

Motivated by the pursuit of a systematic computational and algorithmic of Generative Adversarial Networks (GANs), we present a simple unified non-asymptotic local convergence theory for smooth two-player, which subsumes several discrete-time gradient-based saddle point. The analysis reveals the surprising nature of the off-diagonal term as both a blessing and a curse. On the one hand, this term explains the origin of the slow-down effect in the convergence Simultaneous Gradient Ascent (SGA) to stable Nash equilibria. On the other, for the unstable equilibria, exponential convergence can be proved thanks the interaction term, for four modified dynamics proposed to stabilize GAN: Optimistic Mirror Descent (OMD), Consensus Optimization (CO), Updates (IU) and Predictive Method (PM). The analysis uncovers the connections among these stabilizing techniques, and provides detailed on the choice of learning rate. As a by-product, we present a analysis for OMD proposed in Daskalakis, Ilyas, Syrgkanis, and Zeng 2017 improved rates.