RL-MPC: Reinforcement Learning Aided Model Predictive Controller for Autonomous Vehicle Lateral Control

This paper presents a nonlinear model predictive controller (NMPC) coupled with a pre-trained reinforcement learning (RL) model that can be applied to lateral control tasks for autonomous vehicles. The past few years have seen opulent breakthroughs in applying reinforcement learning to quadruped, biped, and robot arm motion control; while these research extend the frontiers of artificial intelligence and robotics, control policy governed by reinforcement learning along can hardly guarantee the safety and robustness imperative to the technologies in our daily life because the amount of experience needed to train a RL model oftentimes makes training in simulation the only candidate, which leads to the long-standing sim-to-real gap problem–This forbids the autonomous vehicles to harness RL’s ability to optimize a driving policy by searching in a high-dimensional state space. The problem of robustness and constraints satisfaction can be alleviated by using NMPC technique which has proved itself in various industrial control tasks; however, traditional NMPC usually uses one fixed set of parameter matrices in its cost function while the changing path-tracking conditions faced by an autonomous vehicle may require the optimizer to place varying emphasis on different terms of the objective. Therefore, we propose to use a RL model to dynamically select the weights of the NMPC objective function while performing real-time lateral control of the autonomous vehicle (we call this RL-NMPC). The RL weight-search model is trained in a simulator using only one reference path, and is validated first in a simulation environment and then on a real Lincoln MKZ vehicle; the RL-NMPC achieved considerably better performance in lateral tracking during simulation and on-board tests.