September 20, 2025

Maximum Entropy Softmax Policy Gradient via Entropy Advantage Estimation

Key Points

Extending proximal policy optimisation and trust region policy optimisation enhances performance in policy evaluation.
The proposed entropy advantage policy optimisation method simplifies MaxEnt RL application by separating task and entropy objectives.
Empirical evaluations indicate significant improvements in utilisation and generalization across various environments.
MaxEnt RL offers a stable framework, particularly beneficial for discrete action spaces in reinforcement learning tasks.

Abstract

Entropy Regularisation is a widely adopted technique that enhances policy optimisation performance and stability. Maximum entropy reinforcement learning (MaxEnt RL) regularises policy evaluation by augmenting the objective with an entropy term, showing theoretical benefits in policy optimisation. However, its practical application in straightforward direct policy gradient settings remains surprisingly underexplored. We hypothesise that this is due to the difficulty of managing the entropy reward in practice. This paper proposes Entropy Advantage Policy Optimisation (EAPO), a simple method that facilitates MaxEnt RL implementation by separately estimating task and entropy objectives. Our empirical evaluations demonstrate that extending Proximal Policy Optimisation (PPO) and Trust Region Policy Optimisation (TRPO) within the MaxEnt framework improves optimisation performance, generalisation, and exploration in various environments. Moreover, our method provides a stable and performant MaxEnt RL algorithm for discrete action spaces.

Mark Helpful

Bookmark

Relay