Dynamic programming approach to a final-value control system with a random variable having an unknown distribution function

As described in the introduction of this paper, the state vector xn, representing the present state of a sampling control system, is assumed to satisfy a difference equationxₙ₊₁ = T(xₙ, rₙ, υₙ), whereυₙis the control vector of the system subject to random disturbances rn. The random variables rnare assumed to be independent of each other and to be defined in a parameter space in the following manner: Nature is assumed to be in one of a finite number,q, of possible states and each state has its own parameter value, thus specifying uniquely and unequivocally the distribution function of rn. It is further assumed that we are given the a priori probabilityz =(z₁, . . . , zq)of each possible state of nature,i= 1,2, . . . , q,∑min{i=1}max{q} zᵢ=1, zᵢ ≥ 0. Given a criterion of performance, the duration of the process and the domain of the control variable, a sequence of control variables {υₙ} is to be determined as a function of the state vector of the system and time, so as to optimize the performance. The sequential nature of the determination ofυₙis evident here, since the stochastic nature of the problem prevents specification of such a sequence ofυ's as a function of the initial state and time. By means ot the functional equation technique of dynamic programming, a recurrence relation of the criterion function of the process,kₙ(x, z), is derived, wherexis the state variable of the system when there remainncontrol stages. Whenqis taken to be two and the criterion of performance to hexN²with the constraint on the control variables∑min{i=0}max{N-1} υᵢ² ≤ K, where xnis the final state vector of the system, it is shown thatkₙ(x, z)is the formwₙ(z)x²and the optimalυₙ(x, z)is linear inx, as might be expected. The dependence ofkₙ(x, z)andυₙ(x, z)onzis investigated further, andwₙ(z)is found to he concave inz. Explicit quadratic forms forwₙ(z)are obtained in the neighborhood ofz = 0and 1. The optimalυₙ(x, z)is found to be monotonically decreasing inz. When the domain ofυₙis restricted to a finite set of values, as in contactor servo systems, no explicit expressions forkₙ(x, z)andυₙ(x, z)are available. However,kₙ(x, z)is still concave inzand approximately given byzkₙ(x, 1) + (1 - z)kₙ(x, 0). By solving the recurrence relation numerically, this approximation is found to be very good for moderately large n, say 10. This means that if one has explicit solutions forp =p₁andp = P₂, then one can approximate the criterion function for the adaptive case wherePr(p =p₁)= zby a linear function inzas shown above. This provides fairly good lower bound onkₙ(x, z), and in turn serves to determine an initial approximate policy. A concept of suboptimal policy is introduced and numerical experiments are performed to test suboptimal policy suggested from numerical solution. The system behavior under the suboptimal policy is also discussed.