Sequential neural operator transformer for high-fidelity surrogates of time-dependent non-linear partial differential equations

Partial differential equations (PDEs) are fundamental to modeling complex and nonlinear physical phenomena, but their numerical solution often requires significant computational resources, particularly when a large number of forward full solution evaluations are necessary, such as in design, optimization, sensitivity analysis, and uncertainty quantification. Recent advances in artificial intelligence – particularly operator learning – have enabled surrogate models that efficiently predict full-field PDE solutions; however, these models often struggle with accuracy and robustness when faced with highly nonlinear responses driven by sequential input functions. To address these challenges, we propose the Sequential Neural Operator Transformer (S-NOT), an architecture that combines gated recurrent units (GRUs) with the self-attention mechanism of transformers to address time-dependent, nonlinear PDEs. Unlike sequential-deep operator networks(S-DON), which use a dot product to merge encoded outputs from the branch and trunk sub-networks, S-NOT leverages attention to better capture intricate dependencies between sequential inputs and spatial query points. We benchmark S-NOT on three challenging datasets from real-world applications with plastic and thermo-viscoplastic highly nonlinear material responses: multiphysics steel solidification, a three dimensional (3D) lug specimen, and a dogbone specimen under temporal and path-dependent loadings. The results show that S-NOT yields prediction errors up to 4.5 times smaller than S-DON even for data outliers. Furthermore, S-NOT provides an acceleration of 4 orders-of-magnitude compared to traditional finite element method simulations, demonstrating its accuracy and robustness for drastically accelerating computational frameworks in scientific and engineering applications.