Autonomous Berthing Motion Planning for Under-Actuated ASV Based on Imagined Sub-Goals and Soft Actor-Critic

Autonomous surface vehicles (ASVs) face limitations in their motion due to the maneuverability and environmental factors. Effective motion planning and control algorithms are essential for achieving autonomous navigation for these vessels. For the autonomous berthing motion planning problem of under-actuated ASVs, this paper proposes an approach based on hierarchical reinforcement learning algorithm that incorporates imagined sub-goals and soft actor-critic (HRL-ISSAC). Three layers of policies named the top-level policy, sub-goal arrival policy and auto-berthing policy are implemented in the algorithm. Throughout the training process, the top-level policy is designed to continuously generate appropriate sub-goals that are less difficult to reach than the ultimate berthing target status for ASV. Then the agent would train the sub-goal arrival policy by effectively addressing sub-goals. The Kullback-Leibler Divergence is utilized to reduce the distance between the distributions of sub-goal arrival policy and auto-berthing policy, thereby narrowing down the scope of action exploration of auto-berthing policy and accelerating its training speed. The hindsight experience replay mechanism is also employed in our algorithm to deal with sparse reward problem and optimize the sample efficiency. In the testing process the auto-berthing policy is solely applied, thus simplifying the structure of HRL. The simulation results show that HRL-ISSAC successfully achieved autonomous berthing guidance for the agent and the berthing trajectories generated by our algorithm take both the kinetics and dynamic constraints of ASV into consideration, which validate the effectiveness of our algorithm.