Optimal Control Problems of Fully Coupled FBSDEs and Viscosity Solutions of Hamilton--Jacobi--Bellman Equations

In this paper we study stochastic optimal control problems of fully coupled forward-backward stochastic differential equations (FBSDEs). The recursive cost functionals are defined by controlled fully coupled FBSDEs. We use a new method to prove that the value functions are deterministic, satisfy the dynamic programming principle, and are viscosity solutions to the associated generalized Hamilton--Jacobi--Bellman (HJB) equations. For this we generalize the notion of stochastic backward semigroup introduced by Peng Topics on Stochastic Analysis, Science Press, Beijing, 1997, pp. 85--138. We emphasize that when σ depends on the second component of the solution $(Y, Z)$ of the BSDE it makes the stochastic control much more complicated and has as a consequence that the associated HJB equation is combined with an algebraic equation. We prove that the algebraic equation has a unique solution, and moreover, we also give the representation for this solution. On the other hand, we prove a new local existence and uniqueness result for fully coupled FBSDEs when the Lipschitz constant of σ with respect to z is sufficiently small. We also establish a generalized comparison theorem for such fully coupled FBSDEs.