Predicting the Future States of Physicochemical Systems Using Transformer-Based generative Machine Learning Model

Abstract Capturing the time evolution and predicting future kinetic states of physicochemical systems present significant challenges due to the precision and computational effort required. In this study, we demonstrate that the transformer, a machine learning model renowned for machine translation and natural language processing, can be effectively adapted to predict the dynamical state-to-state transition kinetics of biologically relevant physicochemical systems. Specifically, by using sequences of time-discretized states from Molecular Dynamics (MD) simulation trajectories as input, we show that a transformer can learn the complex syntactic and semantic relationships within the trajectory. This enables this generative pre-trained transformer (GPT) to predict kinetically accurate sequences of future states for a diverse set of models and biomolecules of varying complexity. Remarkably, the GPT can predict future states much faster than traditional MD simulations. We show that it is particularly adept at forecasting the time evolution of an out-of-equilibrium active system that do not maintain detailed balance. An analysis of self-attention mechanism inherent in transformers is found to hold crucial role for capturing the long-range correlations necessary for accurate state-to-state transition predictions. Together, our results highlight the ability of transformer based machine learning model in generating future states of physicochemical systems with statistical precision.