DLL: A Risk‐Aware Backup Algorithm for Offline Reinforcement Learning

ABSTRACT Offline reinforcement learning (offline RL) aims to learn an optimal policy directly from a fixed dataset. However, distributional shift between the behavior policy and the current policy can induce out‐of‐distribution actions. Combined with function‐approximation error, this often leads to systematic ‐value overestimation and unstable training. Existing conservative methods mainly shrink the value function as a whole. They describe uncertainty over different actions only in a coarse way. As a result, they still struggle to finely suppress overestimation on out‐of‐distribution actions while keeping high‐quality actions in the data‐support region. To address this, we propose diffusion–LCB–LME (DLL). For a given state, a diffusion policy generates a set of candidate actions near the high‐density region of the dataset. Based on a ‐ensemble, we build a lower confidence bound (LCB) at the action level. This LCB is injected into both the critic target and the actor update, so policy updates prefer “high‐mean, low‐disagreement” safe regions. At the same time, we replace the hard max backup operator with a log‐mean‐exp (LME) operator to smoothly aggregate candidate ‐values. This keeps sensitivity to high‐value actions while reducing target variance and easing out‐of‐distribution overestimation. On the D4RL benchmarks, DLL significantly outperforms several mainstream offline RL methods on tasks with clear distributional shift, such as Adroit and AntMaze. It also matches the performance of diffusion‐QL on MuJoCo tasks. These results show that DLL can suppress out‐of‐distribution overestimation and improve policy returns.