Unbiased fully nonlinear data assimilation: the Stochastic Particle Flow Filter

Nonlinearities in numerical models for the geosciences and in observation operators that map model states to observation space have become so strong that they can no longer be ignored. The particle flow filter (PFF) is a fully nonlinear and efficient sequential Monte Carlo filter that removes the weight degeneracy problem in particle filters by iteratively transporting the equal-weighted particles from the prior to the posterior distribution. The deterministic version of the PPF has been successfully applied to high-dimensional systems and is unbiased in the limit of an infinite number of particles. However, with a small number of particles, the ensemble spread can be biased low, especially in the observed part of the state space. This can be partly alleviated by using a so-called matrix-valued kernel in the algorithm, but the fundamental issue remains. To address this challenge, we propose a novel approach, the Stochastic Particle Flow Filter (SPFF), which includes a Gaussian noise in the Stein Variational Gradient Descent dynamics, the amplitude and covariance of which follow directly from theory. With this additional repulsive force between particles, the SPFF guarantees an unbiased posterior pdf, even with a finite number of particles. We demonstrate the performance of the SPFF using detailed experiments with the 1000-dimensional Loreanz-96 model. Our results demonstrate that SPFF successfully avoids particle collapse of the marginal distributions and accurately captures the evolutions of particles Additionally, and initially unexpectedly, the SPFF exhibits faster convergence than the deterministic PFF and thus improves analysis accuracy compared to the PFF with a matrix-valued kernel at the computational cost. We also show results of its performance on a high-dimensional ocean model demonstrating that we, as a community, are very close to solving the nonlinear data assimilation problem.