Autonomous UAV Navigation in Dynamic Environments with Double Deep Q-Networks

With the rapidly increasing number and complexity of unmanned aircraft systems (UAS), enabling high-density operations becomes the most important goal for UAS operations in congested airspace. However, it is difficult to capture the global environment information such as geolocation of other unmanned aerial vehicles (UAVs) and the steep terrain in real-time. As a result, avoiding dynamic obstacles rather than static ones is challenging. Previous work demonstrates the feasibility of using traditional Q-learning to solve the navigation problem in a static environment, but this method is problematic when facing a dynamic environment because it usually causes the overestimation of action values. To address this challenge, this paper presents a framework based on double deep Q-network with priority experience replay (DDQN-PER) which allows the UAVs to navigate and avoid obstacles in a dynamic environment. The model is built upon convolutional neural networks (CNNs) whose input is raw pixels of the local known environment and whose output is an action after estimating future rewards. We set up multiple experimental scenarios with static and moving obstacles for different tasks which are ranging from single-agent navigation to multi-agent navigation. Then this model is applied to other pre-defined environments, without adjustment of the architecture or learning algorithm, to validate its generalization. Experimental results demonstrate that our proposed models can allow the UAVs reach the goal successfully in new dynamic environments.