Leveraging Probabilistic Optimal Control for Efficient Trajectory Optimization

ABSTRACT This paper discusses two algorithms tailored to discrete‐time deterministic finite‐horizon nonlinear optimal control problems or so‐called trajectory optimization problems. Our key aim is to probe the optimization landscape more efficiently during iterations than traditional gradient‐based approaches do. This is achieved by first reformulating the problem as a risk‐sensitive stochastic optimal control (RSOC) and introducing probabilistic policies. The problem can then be cast as an instance of probabilistic optimal control. In turn this allows us to address the problem using the expectation‐maximization (EM) algorithm which produces a fixed‐point iteration of probabilistic policies that converge to the original optimum. These manipulations facilitate an alternative manner to search the original optimization space without affecting the outcome. In practice, we approximate the probabilistic policies using Gaussian linear affine controllers and rely on sigma‐point uncertainty quantification methods to propagate uncertainty through the system dynamics. The proposed algorithms are structurally closest related to the differential dynamic programming algorithm and related methods that use sigma‐point methods to avoid direct gradient evaluations. However, instead of establishing an ad hoc numerical iteration, a principled recursion is established that provably converges to the true optimum. The algorithms feature improved numerical stability and accelerated convergence as is demonstrated through numerical simulations on different nonlinear systems.