Q-learning-based optimal tracking for uncertain linear discrete-time systems via multitask adaptive optimal feedback control

An online Q-learning scheme is introduced to solve the discrete-time (DT) finite-horizon optimal tracking problem for linear systems using state and output feedback with imperfect system knowledge. To circumvent the reliance on system models, we develop an Adaptive Dynamic Programming (ADP) approach to estimate the optimal Q-function parameters based on real-time input and output data. In our approach, we decompose the explicitly time-varying Q-function parameters into a time-varying basis and time-invariant weight vector with state feedback initially. To relax the state measurements, the controller is significantly extended from the traditional state-feedback formulation to an output-feedback implementation. We integrate an adaptive observer which simultaneously estimates the state vector and unknown system dynamics using input and output measurements because most conventional observers require the system dynamics for good performance. The use of the adaptive observer allows the overall control system to achieve near-optimal performance, significantly outperforming model-based observers with identical initial model mismatch. Finally, the framework is extended to a multi-task control setting, designed to facilitate rapid adaptation across a collection of related systems with differing dynamics. Leveraging concepts from lifelong learning, the Q-function parameters are decomposed into a shared knowledge-base and task-specific parameter vectors. This sparse decomposition structure allows new tasks to be learned efficiently and quickly, benefiting from the knowledge accumulated across previously solved tasks. The resulting scheme minimizes the average cost across multiple systems, demonstrates beneficial knowledge transfer between tasks, and further relaxes the requirement for prior knowledge of the system’s nominal behavior. Simulation results verify the theoretical claims.