June 3, 2024Open Access

Value Improved Actor Critic Algorithms

Key Points

Value-Improved Actor-Critic algorithms enhance performance in various environments, showing a significant improvement over baselines.
Both VI-TD3 and VI-DDPG algorithms demonstrated improvements or matched baseline performance across all Mujoco benchmark tasks.
This analysis employs actor-critic frameworks with dual operators for policy and value updates, enhancing traditional reinforcement learning approaches. Overall effectiveness suggests a promising advancement in the reinforcement learning methodology.

Abstract

Many modern reinforcement learning algorithms build on the actor-critic (AC) framework: iterative improvement of a policy (the actor) using policy improvement operators and iterative approximation of the policy's value (the critic). In contrast, the popular value-based algorithm family employs improvement operators in the value update, to iteratively improve the value function directly. In this work, we propose a general extension to the AC framework that employs two separate improvement operators: one applied to the policy in the spirit of policy-based algorithms and one applied to the value in the spirit of value-based algorithms, which we dub Value-Improved AC (VI-AC). We design two practical VI-AC algorithms based in the popular online off-policy AC algorithms TD3 and DDPG. We evaluate VI-TD3 and VI-DDPG in the Mujoco benchmark and find that both improve upon or match the performance of their respective baselines in all environments tested.

Read Full Paperexternally

Mark Helpful

Bookmark

Relay

View Full Paper