Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift

This is the authors' abstract. We don't add key points for this paper.

Off-policy deep reinforcement learning (RL) algorithms are incapable of solely from batch offline data without online interactions with the, due to the phenomenon known as error. This often due to past data available in the replay buffer that may be quite from the data distribution under the current policy. We argue that off-policy learning methods fundamentally suffer from a shift due to the mismatch between the state visitation of the data collected by the behavior and target policies. This distribution shift between current and past samples can significantly the performance of most modern off-policy based policy optimization. In this work, we first do a systematic analysis of state mismatch in off-policy learning, and then develop a novel-policy policy optimization method to constraint the state distribution. To do this, we first estimate the state distribution based on features the state, using a density estimator and then develop a novel constrained-policy gradient objective that minimizes the state distribution shift. Our results on continuous control tasks show that minimizing this mismatch can significantly improve performance in most popular off-policy policy gradient algorithms.