Toward a nonlinear ensemble filter for high‐dimensional systems

Many geophysical problems are characterized by high‐dimensional, nonlinear systems and pose difficult challenges for real‐time data assimilation (updating) and forecasting. The present work builds on the ensemble Kalman filter (EnsKF), with the goal of producing ensemble filtering techniques applicable to non‐Gaussian densities and high‐dimensional systems. Three filtering algorithms, based on representing the prior density as a Gaussian mixture, are presented. The first, referred to as a mixture ensemble Kalman filter (XEnsF), models local covariance structures adaptively using nearest neighbors. The XEnsF is effective in a three‐dimensional system, but the required ensemble grows rapidly with the dimension and, even in a 40‐dimensional system, we find the XEnsF to be unstable and inferior to the EnsKF for all computationally feasible ensemble sizes. A second algorithm, the local‐local ensemble filter (LLEnsF), combines localizations in physical as well as phase space, allowing the update step in high‐dimensional systems to be decomposed into a sequence of lower‐dimensional updates tractable by the XEnsF. Given the same prior forecasts in a 40‐dimensional system, the LLEnsF update produces more accurate state estimates than the EnsKF if the forecast distributions are sufficiently non‐Gaussian. Cycling the LLEnsF for long times, however, produces results inferior to the EnsKF because the LLEnsF ignores spatial continuity or smoothness between local state estimates. To address this weakness of the LLEnsF, we consider ways of enforcing spatial smoothness by conditioning the local updates on the prior estimates outside the localization in physical space. These considerations yield a third algorithm, which is a hybrid of the LLEnsF and the EnsKF. The hybrid uses information from the EnsKF to ensure spatial continuity of local updates and outperforms the EnsKF by 5.7% in RMS error in the 40‐dimensional system.