Reinforcement Learning-Based Controller With NMPC-Assisted Training for Autonomous Surface Vessels

Abstract For autonomous surface vessels, conventional model-based control strategies face challenges due to the inherent uncertainties and inaccuracies associated with the model. This study investigates the application of reinforcement learning (RL) controllers, specifically using the Deep Deterministic Policy Gradient (DDPG) algorithm, to reduce dependency on model accuracy by enabling controllers to learn through experience. To further enhance RL controller performance, this work integrates Nonlinear Model Predictive Control (NMPC) for assisted training. The methodology involves developing a DDPG-based RL controller for the approximated test vessel’s dynamics and training the RL controller for point tracking. Observations reveal that, while the RL controller successfully learns the control actions required for point tracking, the generated actions lack the smoothness necessary for real-world applications. To address this, NMPC is introduced to assist the RL controller in optimizing control actions during training, yielding results that are more suitable for practical implementation. The findings demonstrate that NMPC assisted DDPG controller can effectively learn vessel dynamics through iterative training, reducing the reliance on precise vessel models. Additionally, the integration of NMPC enhances control action quality, and it has been validated with performance indicators via iterative simulations data. The control variance measuring indicators showed a reduction of an average 60% compared to conventional DDPG controller and the tracking accuracy increase compared to NMPC by 58% on average. This work paves the way for a potential combination of established controllers with RL controller in order to enhance accuracy and performance.