Dynamic-layer transformer-based reinforcement learning for observation-constrained multi-agent roundup scenarios

Abstract To address the challenge of cooperative roundup of maneuvering targets under limited perception, this paper proposes TransMARL, a transformer-based multi-agent reinforcement learning framework for observation-constrained coordination. The roundup task is formulated as a decentralized partially observable Markov decision process (Dec-POMDP), together with a local observation model and a dynamically updated interaction graph. The proposed framework combines a graph feature encoding module with a policy execution module to support decentralized decision-making under partial observability. A task-informed reward function is designed to encourage angular coverage, target approach, formation uniformity, and collision avoidance. In addition, the transformer depth is adaptively adjusted according to the team size as an empirically motivated design choice to balance representational capacity and computational cost. Experimental results in a 2D obstacle-free simulation environment show that, under the evaluated settings, TransMARL achieves competitive and often improved performance relative to the selected baselines, especially under constrained sensing radii. These results suggest that the proposed framework is a practical and scalable approach for cooperative control in observation-constrained multi-agent roundup scenarios, while its broader generalization and formal theoretical characterization remain to be further studied.