Reinforcement learning with embedded knowledge models for multi-agent airspace confrontation

To address the decision-making challenges in multi-aircraft cooperative air combat, this paper proposes a knowledge-model-embedded reinforcement learning framework tailored for multi-agent aerial combat scenarios. The proposed method enhances the QMIX algorithm by integrating prioritized experience replay, an ε-greedy decay strategy, a multi-head attention mechanism, and the weighted QMIX (WQMIX) approach, while embedding domain-specific air combat knowledge into the learning process. These enhancements significantly accelerate training convergence and improve learning efficiency. To optimize the parameters of the improved algorithm (IQMIX), including buffer size, batch size, discount factor (gamma), learning rate (lr), and soft update coefficient (tau), an improved hippo optimization algorithm (HOA) is developed and validated through 23 benchmark functions, demonstrating superior optimization performance. The refined HOA, combined with a surrogate-model-based identification technique, is then employed to determine the optimal parameter configuration for IQMIX. Extensive simulation and ablation experiments in three combat environments show that, compared with the original QMIX algorithm, the knowledge-embedded IQMIX (IQMIX3) achieves a 211% improvement in average reward. Compared with the knowledge-embedded QMIX3 variant, IQMIX3 further improves the average reward by 1.55% while reducing training time to 45.84% of QMIX3. These results demonstrate that the proposed knowledge-model-embedded IQMIX exhibits superior learning efficiency and adaptability in multi-agent aerial combat scenarios.