Nonequilibrium training dynamics and scaling laws of language models

Manuscript and CSV data behind the figures and tables of Xu (2026), Nonequilibrium training dynamics and scaling laws of language models, Research Square, doi:10.21203/rs.3.rs-10529159/v1. Why does the loss of a large language model (LLM) fall as a power law of training data and model size? This work explains the neural scaling laws of LLMs with a stochastic multiscale theory borrowed from turbulence and cosmic structure formation, and tests it on the scale-dependent cross-entropy loss of the Pythia model suite, GPT-2 and SmolLM2. Contents: the manuscript (PDF) and 10 CSV files: the plotted data of Figs 1–9 and the model properties of Table 1. The losses are cross-entropy losses of public checkpoints of the Pythia suite (EleutherAI), GPT-2 and SmolLM2, evaluated by the author. The earlier record of this work is zenodo.org/records/21685713. Major findings (Pythia, GPT-2 and SmolLM2 language models): The scale-dependent loss Lr(r, s), the held-out loss when the usable context is restricted to the last r tokens at training step s, defines a loss spectrum EX = −∂Lr/∂r, the analogue of the energy spectrum of turbulence. Treating learning as a biased random walk in r with waiting time τr ∝ r−γ, the exact Fokker–Planck solution gives EX ∝ r−(1+γ) with an exponential cutoff at a scale ξ; Heaps’ law (αH ≈ 1/2) fixes γ = αH/3 = 1/6, so EX ∝ r−7/6. The measured spectra of converged Pythia and GPT-2 models fall as r−1.10 to r−1.21 over r = 2–32. Training has two phases separated by a critical step s* (about 2000 for Pythia-160M). After s* the context gain ΔLr and the full-context loss L∞ are predicted to change as s+1/10 and s−1/10; Pythia-160M gives s+0.075 and s−0.101 over s = 2000–104. The same theory gives the loss-versus-data exponent αD = γ/(2 − 2γ) = 1/10, close to the empirical 0.095; treating model size N as the effective Langevin time gives αN = λγ/(2 − 2γ), with λ = 3/4 giving 3/40 ≈ 0.075, and the joint form L(N, D) = [(Nc/N)λ + Dc/D]αD. Against model size, L∞ of Pythia 14M–1.4B at the final checkpoint falls as Nnb−0.10 (non-embedding parameters) and Ntot−0.155 (total parameters).