From External Interaction to Internal Inference: An Intelligent Learning Framework for Spectrum Sharing and UAV Trajectory Optimization

Unmanned aerial vehicle (UAV) communication is of crucial importance in realizing heterogeneous practical wireless application scenarios. However, it is susceptible to the severe spectrum scarcity and interference issues since it operates in the unlicensed frequency band. To tackle those issues, a dynamic spectrum sharing UAV network adopting an anti-jamming technique is considered. Two intelligent spectrum allocation and trajectory optimization schemes are designed, capitalizing on the proposed external interaction and internal inference based frameworks. For the first scheme, a novel external interaction based hybrid online-offline multi-agent actor-critic and deep deterministic policy gradient (MA2C-DDPG) framework is proposed taking into account the hybrid characteristics of discrete spectrum allocation and continuous UAV trajectory. As for the second scheme, another novel framework, the deep active inference (DAI) based on internal inference is proposed, which minimizes the internal variational free energy. Moreover, a belief learning based method is exploited to enhance the agents' perception and improve the action selection in the dynamic spectrum sharing environment. Extensive simulation results demonstrate the high efficiency of our proposed schemes. It is shown that our proposed schemes significantly improve the secondary network sum transmission rate compared to various benchmark schemes. Moreover, the proposed MA2C-DDPG and DAI frameworks demonstrate the advantages in improving the training stability and convergence speed.