Investigating Primacy Bias in Multiagent Reinforcement Learning: An Empirical Study

In the past two years, the phenomenon of primacy bias in reinforcement learning has been extensively investigated, sparking discussions within the community regarding the plasticity of reinforcement learning. This study represents the first comprehensive exploration of primacy bias in multiagent reinforcement learning. Building on our previous works, we demonstrate that multi-agent reinforcement learning also encounters the challenge of primacy bias. We then provide a comparative analysis across various settings, including different evaluation methods, the sharing of policy parameters, and the adaptability of decentralized policies. We conducted extensive experiments on multiple multi-agent benchmarks. Our findings reveal specific characteristics of primacy bias in multi-agent learning, showing the difference between them with those in single-agent reinforcement learning. While, we discuss the limitations and challenges encountered when directly applying existing solutions from the reinforcement learning domain to multi-agent scenarios.