Autonomous drifting using simulation-aided reinforcement learning

We introduce a framework that combines simple and complex continuous state-action simulators with a real-world robot to efficiently find good control policies, while minimizing the number of samples needed from the physical robot. The framework combines the strengths of various simulation levels by first finding optimal policies in a simple model, and then using that solution to initialize a gradient-based learner in a more complex simulation. The policy and transition dynamics from the complex simulation are in turn used to guide the learning in the physical world. A method is developed for transferring information gathered in the physical world back to the learning agent in the simulation. The new information is used to re-evaluate whether the original simulated policy is still optimal given the updated knowledge from the real-world. This reverse transfer is critical to minimizing samples from the physical world. The new framework is demonstrated on a robotic car learning to perform controlled drifting maneuvers. A video of the car's performance can be found at https: //youtu.be/opsmd5yuBF0.