The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagrams

We study the gradient-based training of large-depth residual networks (ResNets) from standard random initializations. We show that with a diverging depth L, a fixed embedding dimension D, and an arbitrary hidden width M, the training dynamics converges to a Neural Mean ODE training dynamics. Remarkably, the limit is independent of the scaling of M, covering practical cases of, say, Transformers, where M (the number of hidden units or attention heads per layer) is typically of the order of D. For a residual scale ΘD (αLM), we obtain the error bound OD (1L+ αLM) between the model's output and its limit after a fixed number gradient of steps, and we verify empirically that this rate is tight. When α=Θ (1), the limit exhibits complete feature learning, i. e. the Mean ODE is genuinely non-linearly parameterized. In contrast, we show that α yields a ODE regime where the Mean ODE is linearly parameterized. We then focus on the particular case of ResNets with two-layer perceptron blocks, for which we study how these scalings depend on the embedding dimension D. We show that for this model, the only residual scale that leads to complete feature learning is Θ (DLM). In this regime, we prove the error bound O (1L+ DLM) between the ResNet and its limit after a fixed number of gradient steps, which is also empirically tight. Our convergence results rely on a novel mathematical perspective on ResNets: (i) due to the randomness of the initialization, the forward and backward pass through the ResNet behave as the stochastic approximation of certain mean ODEs, and (ii) by propagation of chaos (that is, asymptotic independence of the units) this behavior is preserved through the training dynamics.