HGRL: A heterogeneous graph-based reinforcement learning framework for robotic swarm applications

Graph neural networks provide a structured approach for representing entities and their relationships in multi-agent systems. In this paper, we propose a heterogeneous graph-based multi-agent reinforcement learning framework, termed HGRL, to support decision-making in robotic swarms. HGRL represents the interactions among agents and environmental entities using heterogeneous graphs. Unlike graph-based MARL methods that retain conventional transition tuples, HGRL encodes state–action transitions as variable-size heterogeneous graph objects and preserves this representation during experience replay and actor-critic optimization. This graph-based formulation allows changes in the numbers and types of agents and environmental entities without restructuring fixed-width experience entries. A graph-based multi-agent soft actor-critic agent is developed to demonstrate HGRL for UAV swarm collaborative navigation, using attention-based graph neural networks to process variable swarm topologies. Under the adopted training setting, the reward and network losses converged within a small number of episodes. Simulation experiments demonstrate successful navigation across different swarm sizes, formations, and planar goal directions. Real-world experiments further demonstrate the transferability of the high-level reference-velocity policy to homogeneous and heterogeneous UAV platforms without fine-tuning. A video of the results is available at https://youtu.be/Ch3ataf1ItQ .