Multi-policy Soft Actor-Critic Reinforcement Learning for Autonomous Racing

Deep reinforcement learning holds immense promise for applications in autonomous driving tasks. For autonomous racing that needs to explore the physical limits of the vehicle in unstructured and dynamic environments, employing reasonable strategies to improve learning efficiency, reduce overall costs, and increase task success rates is of paramount importance. In racing scenarios, strategies primarily manifest as the coordination of throttle, brake, and steering actions taken by the driver. However, existing algorithms have paid limited attention to the relationships among these actions. They often utilize a single policy network to learn the coordination of all actions, leading to low learning efficiency and difficulties in capturing the relationships between actions. This paper proposes a multi-policy Soft Actor-Critic algorithm, which decomposes the complex driving policy into throttle, brake, and steering sub-policies, organizing them in the form of a Bayesian network for the collective training of neural network models. The proposed approach is evaluated in The Open Racing Car Simulator, which is known for its detailed and realistic modeling of cars and tracks. Compared to the state-of-the-art reinforcement learning approaches such as Soft Actor-Critic, Deep Deterministic Policy Gradients and Proximal Policy Optimization, the proposed approach demonstrates faster convergence and improved driving performance. Consequently, the trained controller achieves driving performance comparable to that of an experienced human driver.