Asymptotic Behavior of Stochastic Systems Possessing Markovian Realizations

Markovian stochastic systems of the form Φ k + 1 = F(Φ ₖ ,wk + 1 ) are considered, where w is an independent and identically distributed process and Φ is a Markov chain, both evolving in Euclidean space. A condition called weak stochastic controllability (w.s.c.) is introduced, which is a generalization of the concept of controllability in linear system theory. Two different formulations of stability (boundedness in probability and boundedness in probability on average) are presented, and it is shown that these conditions are equivalent under the w.s.c. condition. These results are related to the ergodic behavior of Φ by a set of results that includes the following: if Φ is bounded in probability and w.s.c., then (i) sample path averages of functions of the state process converge for every initial condition, and (ii) if the stability is uniform then state process probabilities converge to a periodic orbit of probabilities. Finally, if the origin of an undriven system is globally attracting, the linearized system is controllable, and some technical conditions hold, then it is shown that Φ is w.s.c.