Hierarchical Two-Agent Reinforcement Learning for Long-Horizon Energy Optimization and Short-Horizon Navigation in Autonomous Electric Boats

This paper presents a hierarchical two-agent deep reinforcement learning architecture for an electric racing boat, addressing the joint challenges of long-horizon energy management and short-horizon autonomous navigation. The system integrates two independently trained agents: an energy optimization agent based on an importance-weighted actor–learner architecture, selected for its ability to handle long-horizon decision-making and distributed training, and a navigation agent trained with Proximal Policy Optimization, chosen for its training stability. The navigation policy tracks a target speed issued by the energy agent for each track segment. This target speed is the only coupling between the two levels. A curriculum learning strategy is used for both tasks. In a race simulator calibrated on real telemetry, the energy agent reaches the full 34-lap configuration, with a training success rate of 0.51 on average and about 0.72 at the end of training, while the navigation agent reaches its 0.80 evaluation success criterion under a ±5 km/h tolerance. A Unity3D-based environment with hydrodynamic modeling and autopilot software-in-the-loop emulation provides a framework for validation prior to on-water deployment; the closed-loop evaluation of the coupled agents remains future work.