Adaptive Spatio‐Temporal 3D Gaussian Splatting for Scenes with Oscillatory Motion

Abstract Our goal is to reconstruct scenes with stochastic, incoherent motion such as leaves moving in the wind, that can be particularly challenging because of small objects with similar appearance that move independently. Previous dynamic 3D Gaussian Splatting solutions either represent motion implicitly with neural networks achieving good quality but lower framerate, or explicitly with a function, often with higher training times and lower quality. To overcome these limitations, we propose an explicit method that introduces adaptive space‐time densification and smoother optimization. We introduce a new densification approach based on error moments that are used to guide primitive splitting, and we adaptively refine the number of keyframes used based on the variance of error. We observe that dynamic reconstruction from monocular video is hard for standard optimization pipelines. To counter this, we introduce a weighted Adam approach that improves results based on primitive visibility. Finally, to handle the hard case of independent motion of similar‐looking objects, we introduce an image‐driven as‐rigid‐as‐possible regularization. Our method has higher quality than previous explicit solutions, and has significantly higher framerate for rendering.