Reinforcement learning (RL) algorithms enable agents to learn through environmental interaction, mapping states to actions via trial-and-error. Integrating deep learning has expanded their applicability and performance but causes stability issues, mainly from inconsistent sample acquisition and high hyperparameter sensitivity. This paper presents Competitive Swarm Reinforcement Learning (CSRL), a new framework inspired by population-based optimization in evolutionary computing to tackle these problems. In CSRL, a diverse group of agents explores the environment with different strategies, creating a shared pool of varied samples for efficient data use and high-quality experience replay. Also, training multiple strategies with different hyperparameters during execution reduces overall hyperparameter sensitivity and boosts algorithmic robustness. Experiments show CSRL outperforms three baselines in convergence and stability. Compared to its base algorithm, it achieves over 2% higher average returns on 9 out of 10 tasks and a 31.09% improvement on HumanoidBulletEnv-v0 while maintaining sample efficiency. By leveraging contextual state information, CSRL balances exploration and exploitation, surpassing all baselines in rewards, especially in complex scenarios. These results demonstrate that RL systems' performance, stability, and adaptability are significantly improved by competitive swarm dynamics.
Huang et al. (Thu,) studied this question.