Reinforcement Learning for Decentralized Trajectory Design in Cellular UAV Networks With Sense-and-Send Protocol

Recently, the unmanned aerial vehicles (UAVs) have been widely used in real-time sensing applications over cellular networks. The performance of a UAV is determined by both its sensing and transmission processes, which are influenced by the trajectory of the UAV. However, it is challenging for the UAV to determine its trajectory, since it works in a dynamic environment, where other UAVs determine their trajectories dynamically and compete for the limited spectrum resources in the same time. To tackle this challenge, we adopt the reinforcement learning to solve the UAV trajectory design problem in a decentralized manner. To coordinate multiple UAVs performing real-time sensing tasks, we first propose a sense-and-send protocol, and analyze the probability for successful valid data transmission using nested Markov chains. Then, we propose an enhanced multi-UAV Q-learning algorithm to solve the decentralized UAV trajectory design problem. Simulation results show that the proposed algorithm converges faster and achieves higher utilities for the UAVs, compared to traditional singleand multi-agent Q-learning algorithms.