Proximal evolutionary strategy: improving deep reinforcement learning through evolutionary policy optimization

Abstract Evolutionary Algorithms (EAs), including Evolutionary Strategies (ES) and Genetic Algorithms (GAs), have been widely accepted as competitive alternatives to Policy Gradient techniques for Deep Reinforcement Learning (DRL). However, they remain eclipsed by cutting-edge DRL algorithms in terms of time efficiency, sample complexity, and learning effectiveness. In this paper, aiming at advancing evolutionary DRL research, we develop an evolutionary policy optimization algorithm with three key technical improvements. First, we design an efficient layer-wise strategy for training DNNs through Covariance Matrix Adaptation Evolutionary Strategies (CMA-ES) in a highly scalable manner. Second, we establish a surrogate model based on proximal performance lower bound for fitness evaluations with low sample complexity. Third, we embed a gradient-based local search technique within the evolutionary policy optimization process to further improve the learning effectiveness. The three technical innovations jointly forge a new EA for DRL method named Proximal Evolutionary Strategies (PES). Our experiments on ten continuous control problems show that PES with layer-wise training can be more computationally efficient than CMA-ES; our surrogate model can remarkably reduce the sample complexity of PES in comparison to latest EAs for DRL including CMA-ES, OpenAI-ES, and Uber-GA; PES with gradient-based local search can significantly outperform several promising DRL algorithms including TRPO, AKCTR, PPO, OpenAI-ES, and Uber-GA.