Double Actors and Uncertainty-Weighted Critics for Offline Reinforcement Learning

Offline reinforcement learning, also known as batch RL, promises to learn policies effectively from previously collected datasets without exploration. But lack of interaction with the environment continually brings the distribution shift between the learned policy and the behavior policy, which exacerbates the extrapolation errors from out-of-distribution (OOD) actions or states when bootstrapping. Existing methods, whether constraining learned policy close to behavior policy or penalizing the value of OOD actions, are too pessimistic. In this paper, we propose Double Actors and Uncertainty-Weighted Critics (DAUWC), an uncertainty-based actor-critic method exploring the potential of double actors and ensemble critics for efficient moderate pessimistic value estimation. We use double actors to learn a conservative optimistic state value function and back it up into state-action value functions. Meanwhile, we obtain uncertainty estimates for action values by using an ensemble of state-action value networks and penalize the value function based on the estimated uncertainty. Finally, we extract the policy via advantage-weighted behavioral cloning based on the pre-existing framework. Experimental results on D4RL, a standard benchmark for offline RL, show that DAUWC significantly outperforms state-of-the-art methods with higher normalized scores in most of the tasks.