Resource Allocation and Trajectory Design in UAV-Aided Cellular Networks Based on Multiagent Reinforcement Learning

In this article, we focus on a downlink cellular network, where multiple unmanned aerial vehicles (UAVs) serve as aerial base stations for ground users through frequency-division multiple access (FDMA). With user locations and channel parameters inaccessible, the UAVs coordinate to make a decision on resource allocation and trajectory design in a decentralized way. Aiming at optimizing both overall and fairness throughput, we model resource allocation and trajectory design as a decentralized partially observable Markov decision process (Dec-POMDP) and propose multiagent reinforcement learning (RL) as a solution. Specifically, we use parameterized deep Q -network (P-DQN) for the action space comprising both discrete and continuous actions and the QMIX framework is leveraged to aggregate each UAV’s local critics. For fairness throughput optimization, we introduce an entropy-like fairness indicator to the reward to make the total return decomposable. In addition, we further propose a novel distributed learning framework for overall throughput optimization such that each UAV can contribute its local gradient, and model training can be implemented in parallel without need of observation data sharing among the UAVs. Simulation results show that the proposed multiagent RL approach as well as the distributed learning framework are efficient in model training and present acceptable performance close to that achieved by deterministic optimization, which relies on convention optimization techniques with user locations and channel parameters explicitly known beforehand. For fairness throughput optimization, we also show that ground users achieve individual throughputs close to each other, which verifies the effectiveness of the proposed fairness indicator as the reward definition in the RL framework.