AAV Deployment in IoT Networks: A Codebook-Based Reinforcement Learning Approach

This study explores a multiuser massive multiple-input-multiple-output (MU-mMIMO) system that incorporates an autonomous aerial vehicle (AAV) as a decode-and-forward (DF) relay between the base station (BS) and multiple Internet of Things (IoT) devices. The primary goal is to maximize the overall achievable rate (AR) by introducing a novel framework that integrates joint hybrid beamforming (HBF) with AAV deployment in dynamic MU-mMIMO IoT systems. Specifically, considering the geometry-based millimeter-wave (mmWave) channel model for both links, the radio frequency (RF) stages are configured to minimize the number of RF chains by utilizing slow time-varying angular information, while the baseband (BB) stages are developed using reduced-dimension effective channel matrices. Subsequently, deep deterministic policy gradient (DDPG), a reinforcement learning (RL) algorithm with continuous action space, is developed to train the AAV for its deployment. By employing a customized reward function, the RL agent learns an optimal AAV deployment policy capable of adapting to both static and dynamic environments. Then, a novel low-complexity DDPG codebook-based AAV deployment (DDPG-C-AD) is proposed, consisting of an offline agent training phase, and an online AAV deployment prediction to achieve maximum AR. The illustrative results show that the proposed DDPG-C-AD can attain a performance close to the DDPG-based solution in both static and dynamic environments while reducing the runtime by 99%. This makes the DDPG codebook-based solution a promising implementation for real-time online applications in AAV-assisted MU-mMIMO IoT systems.