Dynamic beam hopping based on cooperative MAPPO for LEO satellite communication systems

Due to the cooperative coverage characteristics of Low Earth Orbit (LEO) satellites and the non-uniform traffic demands across beam positions, the flexible and efficient allocation of limited beam and power resources to massive beam locations poses a critical challenge in beam-hopping LEO satellite communication systems. Existing deep reinforcement learning (DRL)-based beam-hopping schemes restrict agents to acquiring state information only within the coverage of a single LEO satellite. To address this, this paper proposes a collaborative framework based on the Multi-Agent Proximal Policy Optimization (MAPPO) to generate dynamic beam-hopping schemes, ensuring both delay fairness and throughput across beam positions in LEO satellite systems. The proposed MAPPO-based dynamic beam-hopping method operates in training and testing phases: During the training phase, each agent, responsible for beam-hopping decisions of a single LEO satellite, learns to collaborate with others via the MAPPO algorithm to maximize system throughput and minimize inter-beam delay disparities. The MAPPO algorithm, with its unique advantage of using clipping to constrain the policy update, prevents significant changes in the policy at each iteration, ensuring more stable policy updates. Critic networks evaluate global states by incorporating the agent's own state - action pairs. In the testing phase, trained agents are deployed distributively on individual LEO satellites to autonomously generate dynamic beam-hopping schemes based on locally observable states. Experiments demonstrate that the proposed MAPPO algorithm effectively addresses non-uniform traffic demands in multi-satellite systems, and simulation results validate its capability to intelligently adapt to beam position requirements, with performance significantly outperforming existing benchmark methods.