An Autonomous Attack Decision-Making Method Based on Hierarchical Virtual Bayesian Reinforcement Learning

In response to the challenges of estimating missile launch timing during close-range unmanned autonomous air combat in the future, this article proposes an autonomous attack decision-making method based on hierarchical virtual Bayesian reinforcement learning (HVBRL). First, a six-degree-of-freedom (6-DOF) high-fidelity aircraft dynamics model, along with missile dynamics and guidance rate models, is constructed. Second, the HVBRL algorithm is introduced, where the low-level algorithm outputs control parameters and the high-level algorithm generates control commands. Given that the number of missile hits on a target under specific conditions follows a binomial distribution, a simple prior knowledge can be introduced through its conjugate prior, the Beta distribution, to avoid prolonged exploration of ineffective areas. Moreover, carrying only a limited number of missiles and predicting the number of hits by multiple virtual missiles in specific states through a neural network circumvent the computational complexity issue associated with carrying an excessive number of missiles. Finally, this article presents the low-level training algorithm, the high-level training algorithm, and the high-level self-play training algorithm. Experimental results show that our method significantly reduces the simulation computational complexity. Compared with the Monte Carlo method carrying 1000 missiles, the simulation speed of the high-level training algorithm is increased by 32.75 times, and that of the high-level self-play algorithm is increased by 23 times. Moreover, the estimated missile hit probability with bias can effectively guide the timing of missile launches in close-range air combat, which has significant implications for intelligent autonomous air combat decision-making and operational analysis.