What type of study is this?

This is a Quantitative Study study.

October 10, 2025Open Access

D2 Actor Critic: Diffusion Actor Meets Distributional Critic

Key Points

D2AC achieves state-of-the-art performance across eighteen challenging reinforcement learning tasks, enhancing learning stability.
The novel policy improvement objective mitigates high variance typical of conventional policy gradients found in reinforcement learning.
A robust distributional critic enhances the model through a blend of distributional RL techniques and clipped double Q-learning.
The evaluation on diverse tasks, including biologically inspired predator-prey scenarios, illustrates significant behavioral robustness.

Abstract

We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach.

D2 Actor Critic: Diffusion Actor Meets Distributional Critic

Key Points

Abstract

Cite This Study

Also Consider

Also Consider