Reinforcement Learning Exploration Strategy Based on Performance Feedback: Asymptotic Convergence Proof and Experimental Validation

To address the limitations of the temperature parameter adjustment mechanism in the Soft Actor–Critic (SAC) algorithm, this paper proposes an exploration-aware SAC (EA-SAC) algorithm. First, we establish a convergence framework for the non-stationary SAC algorithm using a Prešić-type contraction to handle delayed coupling from historical feedback, and we derive the quantitative relationship between the temperature parameter and the Q-function estimation error bound. Second, we construct a policy improvement metric through reward decomposition and design a corresponding adjustment mechanism based on task performance feedback, enabling the agent to autonomously regulate its exploration intensity. Experimental results demonstrate that EA-SAC improves convergence efficiency by approximately 21.4% and 30.9% compared to two SAC variants. Furthermore, in complex environments with dynamic threats, EA-SAC achieves a 79% task completion rate and the highest overall score, significantly outperforming commonly used baseline algorithms. This research provides a novel approach to the exploration–exploitation trade-off problem in maximum entropy reinforcement learning.