Planning the Deployment of Reconnaissance Unmanned Aerial Vehicles Using Reinforcement Learning Methods and Target Distribution Density Function

The experience of combat operations in the Russo-Ukrainian war clearly highlights the growing relevance of employing groups of unmanned aerial vehicles (UAVs) to accomplish a wide range of military tasks. The primary problem that must be solved to ensure their effective deployment is mission planning. In turn, the planning process depends on the UAV group’s control system and the operational conditions. For decentralized control systems, a critical requirement is the capability of onboard autonomous mission planning. A review of the literature indicates that most existing planning algorithms do not account for the specific features of military aerial reconnaissance and target validation, the terrain, and are mostly focused on point-to-point flight optimization. Meanwhile, typical target search planning uses comb, expandable, or free search strategies. However, these approaches are often unsuitable for autonomous route planning by the UAV itself, as coordinates on the map are typically not associated with flight altitude, the instantaneous field of view of onboard sensors, the scale, or image resolution. To enhance the optimization capabilities of autonomous flight planning, this work proposes the use of a Target Distribution Density Function (TDDF). The TDDF is a dynamic two-dimensional mathematical model that describes the conditional relative probability of target presence at various spatial locations. It is created based on terrain data, prior observations, or intelligent assessments that reflect the distribution of potential targets within a particular area or the entire reconnaissance zone. The TDDF enables modeling the environment not as homogeneous, but as a space with varying levels of importance or likelihood of target presence. This study explores the use of the TDDF in reinforcement learning for UAV mission planning. Reinforcement learning, a branch of machine learning, involves training an intelligent software agent (the UAV) to make decisions about the sequence of actions by interacting with the environment to maximize cumulative rewards. In the context of UAV mission planning, the environment modeling process is based on the iterative generation of the TDDF, which – with minor adjustments – serves as the reward function.