Fractional order actor critic and fractional order deep deterministic policy gradient algorithms

The actor-critic and deep deterministic policy gradient algorithms are classic reinforcement learning algorithms. But in many situations, they still have problems of slow convergence and instability. This paper presents a novel integration of Grünwald-Letnikov (G-L) fractional-order derivative into reinforcement learning frameworks, proposing two enhanced algorithms: the fractional-order actor-critic (FOAC) and fractional-order deep deterministic policy gradient (FODDPG). We build the fractional-order policy gradient and fractional-order deterministic policy gradient theorems using the G-L fractional-order derivative to establish the theoretical foundations for fractional-order reinforcement learning. Based on these theoretical advancements, the developed FOAC and FODDPG algorithms implement fractional-order gradient computations to enable more effective optimization. Numerical experiments demonstrate that the new fractional-order algorithms achieve faster convergence and greater stability compared to their traditional counterparts. This work extends the mathematical framework of policy gradient methods, and provides new tools for robust reinforcement learning.