ZeroForward: Zero-Variance Targets for Transformer Language-Model Training

Standard training of an autoregressive language model estimates each conditional next-token distribution from a single observed token, so every gradient carries label noise on top of the noise from sampling contexts. ZeroFold (Jo, 2026a) removed the analogous noise from value learning in reinforcement learning by regressing on exact conditional expectations instead of sampled returns. We transplant this zero-variance-target principle to supervised language-model training. A k-th order conditional table, built in a single pass over the corpus, is treated as the transition probability of a text environment, and the one-hot target at every position is replaced by the exact conditional expectation the table assigns to that state. The table is used only to generate targets and is not part of the model, which is a standard transformer whose feed-forward block takes ZeroFold's residual-cosine form. When the data are k-th order Markov, the replacement leaves the expected gradient and the minimizer unchanged and removes the label-noise term from the gradient covariance exactly; otherwise it introduces a bias that we characterize. Measurements of the gradient noise scale show that exact targets reduce the noise of a training batch by 1.6–1.7× early in training and by 2.2–2.8× late. On Tiny Shakespeare, a 4-layer transformer trained this way reaches the final test loss of the standard baseline with 2.4× less compute (mean over three seeds), table construction included; a 2-layer model saves 2.3–2.6× at matched accuracy (single seed). The same noise accounting explains why the principle yields order-of-magnitude gains in reinforcement learning but several-fold gains here. In ZeroFold both the target variance and the state-sampling variance are eliminated in closed form. In supervised learning only the former can be: we prove that a training objective can read the data through a fixed finite statistic for every parameter value only if the loss is linear in fixed features, so for deep nonlinear models the input-sampling variance arising from the empirical measure cannot be removed by any precomputed statistic.