RCAR: A Reinforcement-Learning-Based Routing Protocol for Congestion-Avoided Underwater Acoustic Sensor Networks

Underwater acoustic sensor networks (UASNs) have attracted much attention due to various aquatic applications. However, UASNs have many specific characteristics, such as high propagation delay, high packet error rate, and low bandwidth, which bring challenges to network congestion control. Furthermore, the point-to-point congestion control algorithms cannot guarantee the end-to-end optimal performance. Therefore, congestion avoidance is an important issue to be considered when designing routing protocols for UASNs. In addition, since the sensor nodes deployed underwater are powered by batteries, which are hard to be replaced or recharged, energy limitation should be taken into account as well. In this paper, we propose a reinforcement-learning-based congestion-avoided routing (RCAR) protocol to reduce the end-to-end delay and energy consumption. With the application of reinforcement learning, the protocol converges to the optimal route from the source node to the surface sink by exploring hop-by-hop. In RCAR, a reward function in reinforcement learning is defined in which congestion and energy are both considered for adequate routing decision. To accelerate the convergence of the algorithm, we introduce a dynamic virtual routing pipe with variable radius, which is related to the average residual energy of the neighbors of the sender node. Moreover, in the cross-layer-information-based RCAR protocol, an information update method based on a handshake in the MAC layer is proposed, which guarantees the optimal routing decision. The simulation results show that the proposed RCAR protocol outperforms hop-by-hop vector-based forwarding protocol (HHVBF), QELAR, and GEDAR in terms of convergence speed, energy efficiency, and end-to-end delay.