Eigen-informed neural ordinary differential equations: Incorporating knowledge of system properties

Using neural ordinary differential equations (ODEs) to model complex systems is still a challenging venture and failing for various reasons, often resulting in convergence to unsatisfactory local minima or unintended termination of the training process because of solver issues. To take a step back, the root of the problem lies in either the complexity of the optimization problem or the lack of data. The negative effects of both aspects can be reduced by incorporating a priori knowledge, a common strategy in Scientific Machine Learning . Especially when modeling physical systems, there is almost always more system knowledge available than is used for training. Examples range from knowledge of generic properties such as “the system is stable” to more quantifiable attributes such as “the system oscillates in a known frequency range”. To close the loop, such knowledge can be used to achieve faster convergence and better generalization. In this paper, we focus specifically on system properties that can be expressed based on eigenvalues, such as (partly) stability, oscillation capability, frequencies, damping, and stiffness. We explain how such properties can be intuitively integrated into training neural ODEs, provide open-source software for this, and finally show in three academic examples that such eigen-informed neural ODEs are able to converge in fewer steps (median up to factor ), generalize better (median up to factor ) and are solvable more efficiently (median up to factor ) compared to pure neural ODEs, and can even reconstruct a given system on undersampled data. • Incorporate a priori knowledge of system properties from systems and control theory in the training process via differentiable eigenvalues. • Regularize stability or stiffness during training of neural ODEs. • Train neural ODEs on undersampled data by incorporation of frequency and damping information. • Improve convergence (up to factor in the median) and generalization (up to factor in the median). • Learn for ODE representations that are faster to solve (up to factor in the median) and require fewer solver steps.