Trusting Machine Learning With Physics: A Fidelity Verification Framework for Complex Systems

Neural networks (NN) are embraced to model complex systems based only on data, but their opacity remains a concern. It is unclear whether skill comes from learning physical dependencies or data memorization, which could lead to catastrophic errors. We present a fidelity verification framework, generalizable across disciplines, revealing NN prediction strategies using dynamical regimes and mechanistic indicators. Using an idealized model of Atlantic Meridional Overturning Circulation with hysteresis behavior and scarcely represented tipping points, we target regime prediction in present day and climate change induced unseen scenarios. We use a hierarchy of architectures, including individual and ensemble multilayer perceptrons, long short‐term memory networks, and transformers, and assess aleatoric and epistemic uncertainty with Shapley Additive Explanations and Layer‐wise Relevance Propagation. We verify that many NNs did learn underlying dependencies for skillful prediction, but also reveal a richness in learning strategy across architectures. Our framework offers a mechanistic pathway to demonstrate fidelity, supporting optimism while stressing the need for verification.