Controlled Language Generation: Token Trees, Fokker--Planck--HJB Limits, and Certificates

We develop a unified theory of controlled language generation whose primary object is the exact autoregressive prefix tree and whose continuous Fokker--Planck--Hamilton--Jacobi--Bellman (FP--HJB) system is a conditional scaling limit. KL-regularized token control yields a nonlinear soft Bellman recursion backward and a prefix-occupancy recursion forward. We prove that the Fr\'echet derivative of the Bellman operator is the controlled token kernel and that its adjoint is exactly the forward occupancy propagator. This operator identity gives an envelope/sensitivity theorem and two non-asymptotic certificates: a uniform Bellman-residual bound and an occupancy-weighted bound that multiplies root-mean-square residuals by explicit distribution-shift factors. The latter is the paper's central architectural result. The forward block estimates the occupancy and coverage factors needed to certify the backward value block; the model therefore prices the reliability of its own certificate under deployment shift. For continuous latent generation we derive the HJB and FP equations, prove the analogous residual-and-coverage certificate, and state the exact compatibility condition G R⁻¹G⁼a/β under which Hopf--Cole linearization turns optimal steering into a Doob transform. We also identify a control-gauge invariance showing that generation data determine the controlled drift and the matrix G R⁻¹G^, not a unique decomposition into control channel and intervention cost. We give a valid entropy-controlled Markov-chain limit, distinguish finite-temperature guidance from support-changing truncation and diffusion-volatility changes, and separate KL-regularized fine-tuning, value-guided inference, exponential resampling, and best-of-n: the first three can target the same Gibbs law, while best-of-n produces a different rank tilt. Distributional objectives lead to a coupled mean-field FP--HJB system with a variational terminal condition. These results induce a four-block Transformer---reference dynamics, value, occupancy/score, and control---whose losses, coverage diagnostics, and failure gates are mathematically linked rather than merely co-trained.