Optimal Control for Continuous-Time Unknown Nonlinear Affine Systems: A Q-Learning Approach

In this paper, to tackle the optimal control problem, we propose a <tex-math notation="LaTeX">Q</tex-math> -Learning approach for continuous-time nonlinear systems without any dynamic information. Primarily, the Hamiltonian and optimum cost functions are utilized to articulate the <tex-math notation="LaTeX">Q</tex-math> -function of continuous-time affine systems. To reduce the dependence of algorithms on system information, a novel <tex-math notation="LaTeX">Q</tex-math> -Learning approach is derived to obtain optimal solutions of nonlinear continuous-time systems without requiring knowledge of either the drift information <tex-math notation="LaTeX">$p(x)$</tex-math> or input gain <tex-math notation="LaTeX">$q(x)$</tex-math> . To implement this approach, critic and actor neural networks can be iterated alternately using an integral reinforcement learning method to estimate the <tex-math notation="LaTeX">Q</tex-math> -function. Furthermore, all signals in closed-loop system are demonstrated to be ultimate uniform bounded (UUB). It is worth noting that there exist rare literatures focused on the optimal control problem of continuous-time nonlinear uncertain systems via the <tex-math notation="LaTeX">Q</tex-math> -Learning for actor/critic networks iteration. Finally, two simulations are used to confirm the effectiveness of the proposed algorithm. Note to Practitioners —Nonlinear continuous-time systems, being ubiquitous in engineering practice, are widely employed due to their versatility and effectiveness. Aiming at such systems, a <tex-math notation="LaTeX">Q</tex-math> -learning approach with optimal feature is proposed to strengthen control efficiency while reduce costs. However, it is well known that accurately capturing all the dynamic information of the system is a formidable task in practical operation. This defect inevitably weakens the feasibility of model-based control algorithms. Since the <tex-math notation="LaTeX">Q</tex-math> -learning algorithm presented in this paper does not require any dynamic knowledge of systems, it is promising enabler in enhancing the effectiveness and flexibility of engineering activities.