Near-optimal estimates for the ᵖ-Lipschitz constants of deep random ReLU neural networks

This paper studies the ᵖ-Lipschitz constants of ReLU neural networks Φ: Rᵈ → R with random parameters for p ∈ [1,∞]. The distribution of the weights follows a variant of the He initialization and the biases are drawn from symmetric distributions. We derive high probability upper and lower bounds for wide networks that differ at most by a factor that is logarithmic in the network's width and linear in its depth. In the special case of shallow networks, we obtain matching bounds. Remarkably, the behavior of the ᵖ-Lipschitz constant varies significantly between the regimes p ∈ [1,2) and p ∈ [2,∞]. For p ∈ [2,∞], the ᵖ-Lipschitz constant behaves similarly to gp', where g ∈ Rᵈ is a d-dimensional standard Gaussian vector and $1/p + 1/p' = 1$. In contrast, for p ∈ [1,2), the ᵖ-Lipschitz constant aligns more closely to g ₂.