Generalized Gradient Emphasis Learning for Off-Policy Evaluation and Control with Function Approximation

Emphatic temporal-difference (TD) learning (Sutton et al. 2016) is a pioneering off-policy reinforcement learning method involving the use of the followon trace. The recently proposed Gradient Emphasis Learning (GEM, Zhang et al. 2020) algorithm is used to fix the problems of unbounded variance and large emphasis approximation error introduced by the followon trace from the perspective of stochastic approximation. In this paper, we rethink GEM and introduce a novel generalized GEM(β) algorithm to learn the true emphasis. The key to the construction of the generalized GEM(β) algorithm is introducing a tunable hyper-parameter β that is not necessarily the same as the discount factor γ to the GEM operator. We then apply the emphasis estimated by the proposed GEM(β) algorithm to the value estimation gradient and the policy gradient, respectively, yielding the corresponding emphatic TD variant for off-policy evaluation and actor-critic algorithm for off-policy control. Finally, we demonstrate empirically the advantage of the proposed algorithms across a range of problems, for both off-policy evaluation and off-policy control, and for both linear and nonlinear function approximation.