Collective robot reinforcement learning with distributed asynchronous guided policy search

Policy search methods and, more broadly, reinforcement learning can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, training a policy that generalizes well across a wide range of real-world conditions requires far greater quantity and diversity of experience than is practical to collect with a single robot. Fortunately, it is possible for multiple robots to share their experience with one another, and thereby, learn a policy collectively. In this work, we explore distributed and asynchronous policy learning as a means to achieve generalization and improved training times on challenging, real-world manipulation tasks. We propose a distributed and asynchronous version of guided policy search and use it to demonstrate collective policy learning on a vision-based door opening task using four robots. We describe how both policy learning and data collection can be conducted in parallel across multiple robots, and present a detailed empirical evaluation of our system. Our results indicate that distributed learning significantly improves training time, and that parallelizing policy learning and data collection substantially improves utilization. We also demonstrate that we can achieve substantial generalization on a challenging real-world door opening task.