The Ternarization Tax and Its Curvature Signature: Training Dynamics of From-Scratch Ternary Language Models

Ternary-weight language models (w ∈ {-1,0,+1}, ≈1.58 bits) are trained with a straight-through estimator (STE), but the dynamics of that training—and the loss geometry it lands in—are not well characterized. We study from-scratch ternary quantization-aware training (QAT) against an architecturally-identical full-precision (FP) twin on identical seeded batches, at a toy scale (3 seeds) and across a 26–206M scale ladder (S/M/L, 3 seeds each). Findings, mostly corrections of intuitions in the literature. (1) A small, reproducible ternarization tax (0.117±0.013 nats; ≈+12% perplexity)—and the latent weights do not organize toward the lattice (%zeros pinned at 0.31; a null we explain via absmean scale-invariance). (2) The "sharper minimum" is a reparameterization artifact, not geometry. The raw Hessian trace (Hutchinson) of the realized ternary network is 2.1× the FP twin's (λ_max 1.8×), sign-consistent over seeds—but raw parameter-space curvature is not reparameterization-invariant, and the realized ternary weights operate at a smaller norm. Under a pre-registered scale-invariant control—filter-normalization at a matched training loss, 5 seeds—the ratio collapses to 0.99× (the raw ratio reproduces at 2.07×): the sharpness signal does not survive, and the two minima are comparably curved. What is genuinely larger for the ternary model is the Fisher (optimizer-preconditioned) curvature (≈2×)—the excess lives in the optimization surface the STE descends, not the invariant loss geometry. The ternary model also has a no larger (if anything smaller) train–val gap, so curvature and generalization dissociate. (3) There is a clean U-shaped optimum in the 16-bit→ternary transition step t*: an FP warm-up for half of training before switching to QAT halves the tax (0.057 vs. 0.116), while pure post-training quantization (the toy t*=1 endpoint, no QAT) is catastrophic (1.31 nats, ≈23× the best t*)—but the literal "weight-mode-emergence" mechanism is refuted, and the optimum's location is schedule-dependent. (4) A soft, differentiable quantizer is costlier than STE. (5) At scale, the tax shrinks (0.194→0.103); the raw curvature gap grows S→M then saturates, but—per the control above—raw curvature is parameterization-dependent, so we report it as a descriptive signature, not a geometric claim. The robust, falsifiable objects are the tax, its warm-up dependence, the PTQ gap, and the directions, not the raw-curvature magnitudes.