Integration of Decision-Making and Motion Planning for Autonomous Driving Based on Double-Layer Reinforcement Learning Framework

Autonomous driving involves multi-timescale and multi-objective tasks coupled with long-term driving decision and short-term motion planning. However, existing studies tend to investigate them separately or assume they were synchronous and instantaneous, failing to deeply interpret their interconnections in autonomous driving. To address it, this study explored driving decision and motion planning in an integrated manner, and proposed a double-layer reinforcement learning (RL) framework to couple and optimize these two modules. Specifically, on the upper layer of the framework, a trajectory-level reward function associated with safety, efficiency, comfort and decision execution was proposed, and a decision-making model was established based on value function-based deep reinforcement learning (DRL) algorithms. Then, the reward function of the upper layer was taken as the objective function of the lower layer with relevant state constraints. The model predictive control (MPC) method was used to derive the optimal maneuver sequence guided by driving decisions, and the performance evaluation of motion planning was fed back to the decision-making for DRL parameter optimization. Accordingly, a closed-loop mechanism was developed consisting of forward decision-making output guidance and backward motion planning feedback optimization. Then, two-lane and three-lane interactive scenarios were set for framework training and testing. Last, comparative experiments were conducted using NGSIM dataset. The results demonstrated the effectiveness and the enhanced driving performance of the proposed framework in contrast to four benchmarks.