Comparing Machine Learning Methods for Dynamical Systems

We explore the task of learning the dynamics of a system from observed data without prior knowledge of the laws governing the system. Our extensive simulation study focuses on ordinary differential equation (ODE) problems that are specifically designed to reflect key aspects of various machine learning tasks for dynamical systems - namely, chaos, complexity, measurement uncertainty, and variability in measurement intervals. The study evaluates a variety of methods, including neural ODEs, transformers, Gaussian processes, echo state networks, and spline-based estimators. Our results show that the relative performance of the methods tested varies widely depending on the specific task, highlighting that no single method is universally superior. Although our research is predominantly in low-dimensional settings, in contrast to the high-dimensional nature of many climate science challenges, it provides insightful comparisons and understanding of how different approaches perform in learning the dynamics of complex systems.