Contractivity of neural ODEs: an eigenvalue optimization problem

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ODEs. When looking at contractivity properties of a one layer weight-tied neural ODE u̇(t)=σ(Au(t)+b) (with u,b ∈ Rⁿ, A is a given n × n matrix, σ : R → R⁺ denotes an activation function and for a vector z ∈ Rⁿ, σ(z) ∈ Rⁿ has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type $D A$, where D is a diagonal matrix such that diag(D) ∈ σ'( Rⁿ). Specifically, given a real number c (usually $c=0$), the problem consists in finding the largest positive interval χ⊆ [0,∞) such that the logarithmic norm μ(DA) ≤ c for all diagonal matrices D with Dᵢᵢ∈ χ. We propose a two-level nested methodology: an inner level where, for a given χ, we compute an optimizer D⁽χ) by a gradient system approach, and an outer level where we tune χ so that the value c is reached by μ(D⁽χ)A). We extend the proposed two-level approach to the general multilayer, and possibly time-dependent, case u̇(t) = σ( Aₖ(t) … σ ( A₁(t) u(t) + b₁(t) ) … + bₖ(t) ) and we propose several numerical examples to illustrate its behaviour, including its stabilizing performance on a one-layer neural ODE applied to the classification of the MNIST handwritten digits dataset.