Relative Entropy Regularized Policy Iteration

We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function; ii) policy improvement via the estimation of a local non-parametric policy; and iii) generalization by fitting a parametric policy. Each step can be implemented in different ways, giving rise to several algorithm variants. Our algorithm draws on connections to existing literature on black-box optimization and 'RL as an inference' and it can be seen either as an extension of the Maximum a Posteriori Policy Optimisation algorithm (MPO) Abdolmaleki et al., 2018a, or as an extension of Trust Region Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) Abdolmaleki et al., 2017b; Hansen et al., 1997 to a policy iteration scheme. Our comparison on 31 continuous control tasks from parkour suite Heess et al., 2017, DeepMind control suite Tassa et al., 2018 and OpenAI Gym Brockman et al., 2016 with diverse properties, limited amount of compute and a single set of hyperparameters, demonstrate the effectiveness of our method and the state of art results. Videos, summarizing results, can be found at goo.gl/HtvJKR .