Vision based material learning through non-rigid deformations

Incorporating physics into the capture and prediction of object trajectories has long been a focus in computer vision and graphics, with broad applications in gaming, fashion, and engineering. Recent advances demonstrate the effectiveness of using physics-based loss functions to enhance the accuracy of trajectory estimation. In this work, we introduce a novel workflow consisting of Gaussian-based capture and interpolation of non-rigid media and a differentiable finite element method (FEM) simulation framework to infer constitutive parameters of non-rigid objects from deformation sequences. Our approach accurately recovers these meaningful physical properties from a single deformation sequence, without requiring prior knowledge of the object's material characteristics or the surrounding scene. To support evaluation and benchmarking within this domain, we also release synthetic and real-world data comprising 4D deformation sequences of non-rigid objects, complete with ground-truth physical parameters and collider trajectory annotations.