Selective Imitation Enhanced Deep Reinforcement Learning for AAV Navigation and Obstacle Avoidance With Sparse Rewards

Deep reinforcement learning (DRL) has emerged as a promising solution for autonomous operations of autonomous aerial vehicles (AAVs) in unknown environments. However, learning to navigate and avoid obstacles under sparse reward settings remains challenging. In this article, we propose an end-to-end learning approach that synthesizes imitation learning with DRL for AAV navigation and obstacle avoidance. Specifically, we formulate this problem as a partially observable Markov decision process with sparse rewards and learn an end-to-end policy that maps imperfect sensor data to control signals. To efficiently optimize the policy under the sparse reward setting, we propose the selective behavior cloning enhanced actor-critic (SBCAC) algorithm. By integrating an experience filter and a Q-value based action selector to selectively mimic an artificial potential field based non-expert policy, our approach significantly improves the learning performance and sample efficiency. Extensive simulations with fixed-wing and multi-rotor AAVs in different scenarios demonstrate that SBCAC achieves an average improvement of up to 16.07% in success rate, a 72.46% reduction in crash rate, and a 94.62% reduction in stray rate compared to the state-of-the-art selective imitation baseline. Furthermore, hardware-in-the-loop and physical experiments validate the effectiveness of our approach, showing its potential for practical applications in complex environments.