Existence of Optimal Stochastic Control Laws

We give an approach to optimal control of systems described by stochastic functional-differential equations of Itô’s type : $dx(t) = f(t,x,u(t,x))dt + dw(t))$, 0 t 1, with a cost functional k(u) = E∫₀¹ c(t,x,u(t,x))dt to be minimized. Here w( · ) is Brownian motion, f and c are nonanticipative functionals describing system dynamics and cost rate respectively, and u(t, · ) is a causal control law, to be chosen, taking values in a space Γ of control points, and depending at t at most on given information about the past \ x(s),s t\. A key role is played by the information available for control. This is described by giving, for each t, a sub-σ-algebra Gₜ of the σ-algebra Sₜ over the continuous functions $C = C[0,1]$ generated by sets \ y:y(s) ∈ A\ with 0 s t and A Borel in Rᵈ. Sₜ is the σ-algebra corresponding to knowing the whole past of the trajectory prior to t. The set U of admissible control laws consists of functions u:[0,1] × C → Γ which are Lebesgue in t, and Gₜ-measurable in y for each t. There is a σ-algebra G over [0,1] × C such that admissibility is equivalent to G-measurability. Our formulation is based on a result of Girsanov (Teoriya Veroyatnostei, 5 (1960), p. 285) : For φ a nonanticipative functional of Brownian motion w, the transformed measure d P = exp ξ (φ )dP with \[ξ (φ ) = ∫_0^1 {φ \, dw - 1/2} ∫_0^1 {| φ |^2 dt} \] makes the functions w( · ) - ∫₀^ · φ \, dt a Wiener process, provided Eexp ξ (φ ) = 1. This result suggests and justifies taking, for the solution x( · ) of the system equations, the process determined by Girsanov’s device with φ = f(t,w,u(t,w)); in this case we say that u attains the density exp ξ (φ ). The control problem is reformulated as a search for admissible u that achieve inf u ∈ U Eexp ξ (φ )∫₀¹ cdt. That is, with each u ∈ U we associate, as the solution of the system equations to be considered for the purpose of our minimization of k( · ), the functions w( · ) under the measure exp ξ (φ )dP, with the justification that under this measure \[w(t) - ∫_0^t f(s,w,u(s,w))ds\] is a Wiener process. Novelty of the approach lies in these features : (i) control is closed loop; (ii) admissible controls need not be smooth; (iii) the Radon–Nikodym derivative used by Girsanov directly gives a measure corresponding to a solution of the system equations. As a principal result, we prove that if Γ is compact metric, if $f(t,y,u)$ grows at most linearly with $y(t)$, and if Gₜ = Sₜ (i.e., if the whole past is available for control), then the set of densities \ exp ξ (φ ):φ = f(t,w,u(t,w)),u ∈ U\ attainable by the admissible control laws is convex, and there exists an optimal control law u^ * ∈ U achieving inf u ∈ U Eexp ξ (φ )∫₀¹ cdt.