Data Efficient Deep Reinforcement Learning With Action-Ranked Temporal Difference Learning

In value-based deep reinforcement learning (RL), value function approximation errors lead to suboptimal policies. Temporal difference (TD) learning is one of the most important methodologies to approximate state-action ( <tex-math notation="LaTeX">Q</tex-math> ) value function. In TD learning, it is critical to estimate <tex-math notation="LaTeX">Q</tex-math> values of greedy actions more accurately because a more accurate target <tex-math notation="LaTeX">Q</tex-math> value enhances the estimation accuracy of <tex-math notation="LaTeX">Q</tex-math> value. To improve the estimation accuracy of <tex-math notation="LaTeX">Q</tex-math> value, we propose an action-ranked TD learning method to enhance the performance of deep RL by weighting each TD error according to the rank of its corresponding state-action pair's value among all the <tex-math notation="LaTeX">Q</tex-math> values on a state. The proposed method can provide more accurate target values for TD learning, making the estimation of the <tex-math notation="LaTeX">Q</tex-math> value more accurate. We apply the proposed method to a representative value-based deep RL algorithm, and results show that the proposed method outperforms baselines on 31 out of 40 Atari games. Furthermore, we extend the proposed method to multi-agent deep RL. To adaptively determine the hyperparameter in action-ranked TD learning, we propose a meta action-ranked TD learning. A series of experiments quantitatively verify that our methods outperform baselines on Atari games, StarCraft-II, and Grid World environments.