Sharper, Not Flatter: The Training Dynamics of From-Scratch Ternary Language Models

Ternary-weight language models (w ∈ {-1,0,+1}, ≈1.58 bits) are trained with a straight-through estimator (STE), but the dynamics of that training—and the loss geometry it lands in—are not well characterized. We study from-scratch ternary quantization-aware training (QAT) against an architecturally-identical full-precision (FP) twin on identical seeded batches, at a toy scale (3 seeds) and across a 26–206M scale ladder (S/M/L, 3 seeds each). Findings, mostly corrections of intuitions in the literature. (1) A small, reproducible ternarization tax (0.117±0.013 nats; ≈+12% perplexity)—and the latent weights do not organize toward the lattice (%zeros pinned at 0.31; a null we explain via absmean scale-invariance). (2) Ternary minima are sharper, not flatter: the full Hessian trace (Hutchinson estimator) is 2.15× larger than the FP twin's (and λ_max 1.84×), sign-consistent over all 3 seeds—refuting, at this scale, the natural expectation (implicit in quantization-as-regularization readings and in methods that explicitly seek flat minima for quantized nets) that ternary QAT lands in flatter minima. The sign of the effect (ternary sharper) is robust across seeds; its magnitude is parameterization-dependent, since ternary and FP are different parameterizations. Yet the ternary model has a smaller train–val gap in all 3 seeds, so sharpness and generalization dissociate. (3) There is a clean U-shaped optimum in the 16-bit→ternary transition step t*: an FP warm-up for half of training before switching to QAT halves the tax (0.057 vs. 0.116), while pure post-training quantization (the toy t*=1 endpoint, no QAT) is catastrophic (1.31 nats, ≈23× the best t*)—but the literal "weight-mode-emergence" mechanism is refuted, and the optimum's location is schedule-dependent. (4) A soft, differentiable quantizer is sharper and costlier than STE. (5) At scale, the tax shrinks (0.194→0.103), while the curvature gap grows S→M then saturates (always >1) and layer-0's relative sensitivity dominance grows (though its absolute damage falls). The dynamics are a pilot-scale characterization; the robust, falsifiable objects are the directions, not the magnitudes.