Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

Deep reinforcement learning algorithms require large amounts of experience to an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of, several major challenges preclude their practicality. Current rely heavily on on-policy experience, limiting their sample efficiency. also lack mechanisms to reason about task uncertainty when adapting to new, limiting their effectiveness in sparse reward problems. In this paper, address these challenges by developing an off-policy meta-RL algorithm that task inference and control. In our approach, we perform online filtering of latent task variables to infer how to solve a new from small amounts of experience. This probabilistic interpretation posterior sampling for structured and efficient exploration. We how to integrate these task variables with off-policy RL algorithms achieve both meta-training and adaptation efficiency. Our method outperforms algorithms in sample efficiency by 20-100X as well as in asymptotic on several meta-RL benchmarks.