Reward Specifications in Collaborative Multi-agent Learning: A Comparative Study

Reinforcement learning is a prominent learning paradigm that seeks to maximize cumulative rewards over time. Nevertheless, some real-life problems often exhibit inherent sparsity in rewards, which pose difficulties for standard reinforcement learning algorithms in efficiently learning optimal policies without frequent feedback. In multi-agent environments, reward specifications play a crucial role in collaborative learning by designing reward structures that guide agents toward desired behaviors and effectively addressing the challenge of sparse rewards. This paper presents a new study that explores the impact of reward specification techniques on collaborative learning in multi-agent environments. In our experiments, we use state-of-the-art multi-agent reinforcement learning (MARL) algorithms, which have been proven to be effective under dense reward environments, along with different reward specifications with a focus on evaluating their performance under sparsity settings in a variety of environments, including discrete and complex scenarios. In addition, we provide in-depth insights on how diverse factors, such as task nature and information availability, influence the reward specification's impact concerning agent learning and coordination. To assess these aspects, we examine the average team rewards and convergence speed. The results highlight the importance of reward specifications, aiding researchers and practitioners in selecting effective techniques for various real-world collaborative problems.