Emergent collaborative scheduling in satellite swarms: A distributed multi-agent reinforcement learning approach with adaptive experience evolution

Autonomous collaboration among multiple Earth observation satellites is essential for improving the performance of satellite swarms. However, this task is challenging because of limited onboard computing resources and unstable inter-satellite links, particularly when global coordination is required under communication constraints. Traditional centralized optimization methods often adapt poorly to dynamic observation requests, whereas existing multi-agent approaches may incur communication overhead that hinders practical deployment. To address this issue, we propose a distributed scheduling framework that enables emergent global coordination through local agent interactions and formulates the problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). Specifically, we develop L-QTRAN, a multi-agent reinforcement learning algorithm that enables satellites to coordinate based on local observations with limited communication. To alleviate the non-stationarity inherent in dynamic swarm environments, we introduce an Adaptive Experience Evolution (AEE) mechanism. This mechanism follows the survival-of-the-fittest principle by selectively retaining elite transition experiences, thereby accelerating the development of cooperative strategies. Furthermore, a Random Segment Sampling (RSS) technique is introduced to improve training stability and convergence in high-dimensional solution spaces. Extensive experiments demonstrate that the proposed approach outperforms the baseline methods under weak-communication conditions and achieves sub-second response times, highlighting its potential for autonomous satellite swarms operating in dynamic environments.