Guaranteed inference for probabilistic programs: a parallelisable, small-step operational approach

In the context of probabilistic programming languages, we put forward an approach to formal semantics and sampling-based inference with guarantees, centered on an action-based language equipped with a small-step operational semantics. We argue that this choice offers benefits in terms of clarity and effective, vectorized implementations. In measure-theoretic terms, a product of Markov kernels is used to formalize the small-step operational semantics. A trace semantics is also introduced based on a probability space of infinite sequences, along with a finite approximation theorem, relating the exact semantics to a truncated-execution semantics. This result directly leads to a sampling algorithm with guarantees, that can be efficiently SIMD-parallelized. Experiments conducted with an implementation based on TensorFlow show that our approach compares very favourably to state-of-the-art tools for probabilistic programming and inference.