Initially Excited Asynchronous Reinforcement Learning Control With Monotonicity and Stability

This article addresses the existing reinforcement learning (RL) control issues of continuous-time Markov jump systems (MJSs), including their synchronous iteration structure using nonlatest updates, nonmonotonic convergence, and requirement of initial admissible control and all-time persistent excitation (PE) condition. We propose advanced model-based and model-free RL algorithms that: 1) have asynchronous decoupled Lyapunov iteration equations to approximate the optimal control solutions using the latest updates; 2) determine the initial admissible control policy (IACP) and initial value function matrix without relying on engineering experiences; and 3) relax PE condition with a milder initial excitation (IE) condition. Rigorous theoretical analyses are provided to establish the monotonic convergence to the optimal control solution and the closed-loop stability at each iteration. Finally, the simulation and comparison results verify the proposed algorithms.