Optimal Control of Continuous Time, Markov, Stochastic Systems†

The control problem considered is that of a non-linear, time varying plant which is disturbed by, or must follow, some stochastic process. Further, the system of plant plus inputs must be adequately represented by a mathematical model whoso probability function description satisfies the Markov condition. The control is to be chosen, within allowed limits, so as to optimize a given performance index. Using concepts based on Bollman's Dynamic Programming, partial differential equations are set up whoso solution yields the value of the performance index, and the optimal control settings over so mo prescribed time interval. A discussion shows that continuous observation of the system is a theoretical necessity, hi contrast to the deterministic situation. Usually the observations will entail feedback. For the special class of problems with linear plant dynamics, quadratic performance indices, and Gaussian random components, all with time varying parameters, the partial differential equations are reduced to a sot of ordinary differential equations, well suited to machine solution. Because of its simplicity, this method supplants previous solutions to these special problems, based on complex variable and integral equation techniques.