Adaptive Adjustment of Advantage Estimation for Robot Control Using Reinforcement Learning

In robotics control experiments, the balance between exploration and exploitation, as well as the accuracy of the advantage function estimation, are crucial factors that affect the effectiveness of policy optimization methods. To overcome these challenges, this paper proposes an adaptive adjustment of advantage estimation based on the policy loss and policy entropy algorithm (A3E-PLE), which can improve the exploratory capabilities of the proximal policy optimization (PPO) algorithm. Specifically, on the one hand, the policy loss is adjusted using a Gaussian distribution policy entropy to mitigate randomness and separate policy improvement from random noise, thereby improving exploration efficiency. On the other hand, to adapt flexibly to various training scenarios and further enhance the accuracy of advantage function estimation, the policy loss is incorporated into the advantage function estimation. This enables the algorithm to adaptively adjust according to changes in the strategy. Finally, the proposed reinforcement learning (RL) framework was validated using robot control simulations and complex decision-making environments. It is shown that A3E-PLE achieves higher learning efficiency and greater rewards compared to traditional generalized advantage estimation.