Recurrent Neural Networks and (Time-Uniform) Universal Approximation of Bayesian Filters

We consider approximations of the Bayesian optimal filtering problem, e.g., estimating some conditional statistics of a latent time-series signal from an observation sequence. Classical approaches often rely on the use of assumed or estimated transition and observation models. Instead, we formulate a generic recurrent neural network framework and seek to learn directly a recursive mapping from observational inputs to the desired estimator statistics. The main focus of this article is on the approximation capabilities of this framework. We provide approximation error bounds for filtering in general noncompact domains. Importantly, we also consider strong time-uniform approximation error bounds that guarantee good long-time performance, i.e., we show under certain conditions that filter approximation errors do not accumulate unbounded over time. We discuss and illustrate a number of practical concerns and implications of these results.