Digital twin based DDPG reinforcement learning for sum-rate maximization of AI-UAV communications

Abstract Construction of wireless infrastructure using unmanned aerial vehicle (UAV) can effectively expand the coverage and support high-density traffic of next-generation communication systems. Designing wireless systems including UAVs as aerial base stations (ABSs) is a challenging task, due to the mobility of ABSs causing time-varying nature of environmental surroundings and relative propagation paths to user equipment (UE) devices. Therefore, it is essential to have an accurate estimate of the channel for varying positioning of the UAVs. In this paper, we propose to adopt a digital twin based performance evaluation procedure for wireless systems including ABSs, providing enhanced accuracy of channel modeling for specific target deployment areas. Using ray-tracing channel models reflecting detailed building and terrain information of the transmission environment, an UAV position optimization algorithm based on reinforcement learning is presented. By utilizing deep deterministic policy gradient (DDPG), the proposed algorithm calculates the overall throughput in the digital twin and determines the desired states of the UAV. Performance evaluation results demonstrate the trajectory training ability of the algorithm and the performance advantage of the system with a reduced amount of shadow area compared to those with ground base stations (GBSs).