Adaptive Optimal Control of Unknown Stochastic Discrete‐Time Linear Systems

The optimal control with an ergodic cost function is investigated for a class of stochastic discrete‐time linear systems with unknown system coefficient matrices. We present an online value iteration‐based stochastic adaptive dynamic programming algorithm to obtain the approximation sequence of the optimal state feedback control gain and rigorously prove that the approximation sequence converges to the optimal state feedback control gain under some reasonable conditions. In the framework of direct adaptive control, the diminishing excitation technique is introduced for an unknown stochastic discrete‐time state space model, and we prove that the adaptive controller, which is given by real‐time computation, is optimal. In contrast to existing related results, our method does not need to collect enough data to solve the Bellman equation in each iteration and avoids the requirement to estimate means and variances of controllers or states. A numerical simulation is provided to evaluate the efficacy of the presented approach.