Offline reinforcement learning (RL) demonstrated remarkable performance in learning valid policies by benefiting from high-quality offline datasets. However, collecting such a dataset is labor-intensive, especially for humanoid locomotion. For this reason, many data augmentation techniques have been proposed to improve the quality of offline datasets through noise injection or data synthesis. However, existing data augmentation methods are noise-sensitive, resulting in limited capability in complex robotic environments. To address these issues, we propose guided amplified learning with Lipschitz constraint (GALC), a novel trajectory augmentation method that employs the reward-amplification-guided conditional diffusion model for noise-insensitive data augmentation. Specifically, we introduce a local Lipschitz continuity constraint to regulate the reverse denoising process from the offline dataset. Consequently, the exploration of the diffusion model can be restricted within the local continuity region of the original dataset, thereby generating high-reward trajectories. Moreover, the generated trajectories are also enforced to be noise-insensitive to perturbations, thus enjoying robustness. Notably, our proposed method can prevent the generation of unsafe actions that do not align with the environment dynamics. Extensive experiments on sparse reward scenarios and high-dimensional robotic tasks show that our proposed GALC achieves significant improvements in both the augmented trajectories and policy performance.
Lin et al. (Thu,) studied this question.