Answers are grounded in this paper and cite their sources.
The theory of neural networks lacks a single quantity that is defined for every architecture and training procedure, measurable on real pipelines, and sufficient to state what a network can and cannot learn. We argue that such a quantity exists: the implicit code length, the cumulative log loss incurred when each example is predicted before it is trained on. We develop it into a self-contained theory with complete proofs: a conservation law (for every learner and every input set the code lengths of all labelings satisfy Kraft's equality, so inductive bias is a fixed budget that can be moved but not increased); a representation theorem (every learner is the Bayes predictor of its own implicit prior); a compression identity (expected code length equals the sum of test risks along the learning curve), which constrains plateaus, scaling exponents and double-descent bumps; exact accounting for single-pass training, for symmetries of architecture and initialization (including a new corollary: breaking a symmetry moves the tax inside the orbit but cannot remove it), for pretraining and for in-context learning. We state three falsifiable conjectures and a protocol to test them, and report experiments with real numbers. Earlier exact enumerations at $n=10$ and $n=16$ and MNIST code lengths are reproduced in part and extended by four new experiments. (i) Across standard, NTK, maximal-update (μP) and step-rescaled parametrizations, exact priors on all 2¹² labelings show that the concentration of the prior with width reported earlier is a property of a step that grows with width: in μP the entropy is flat (+0.005±0.009 nats per doubling) and the prior converges (KL(Q2H\|QH)∝ H-1.10±0.15), while the revised conjecture that entropy is a function of function-space movement alone is refuted (0.82±0.11 nats between μP and NTK at equal movement). (ii) A centred finite network approaches its neural-tangent-kernel code as H-0.97±0.08 in Kullback--Leibler divergence; beyond width about one hundred the departure is the randomness of the kernel, not feature learning; and infinite-width gradient descent stays $0.37$ nats ($4.5%$ of nln2) from the best tempered Gaussian-process Bayes code. (iii) Sample-wise double descent of a ridgeless random-features classifier is a burst of code rate at the interpolation threshold whose price is $3.9$--$176$ nats (median). (iv) On sparse parities the plateau of single-pass SGD costs 1,495--8,500 nats where the Bayes mixture over the class pays at most $7.04$, and the price is not monotone in width. Kraft's equality holds to 3.3×10⁻¹⁶ on all $313$ new exact codes. The six further experiments of the programme are reported as well. (v) The in-context identity holds to 10⁻¹⁵ on a four-task family; the unavoidable term is $0.752$ nats (H(π)=1.280), and a network pretrained on one task costs up to $15.24$ nats on the opposite one against $6.83$ from scratch. (vi) Exact priors of convolutional, residual and attention architectures on translation orbits fall into two clusters $5$--$15$ nats apart; translation-invariant learners put $0.896$--$0.982$ of their mass on invariant labelings (MLP: $0.016$), almost all of it on constant ones. (vii) On the scikit-learn digits, block-wise code lengths follow power laws in n with exponents that order the pipelines as on MNIST, and code-length ratios over $30$ tasks stay within a factor $1.38$. (viii) Two presentation orders of the same ten examples give priors $0.44$--$0.77$ nats apart, as much as two seeds. (ix) An asymmetric initialization redistributes the symmetry tax inside every orbit, as the corollary predicts, and makes a naming cheaper than its tax for only $2$ of some ten million labelings. (x) On the stored $n=12$ priors, multiplicative invariance holds labeling-wise with a constant of at most $3.7$ for width-stable parametrizations and fails for a fixed-step standard one ($4.1$ at width $64$, $8.1$ at $256$); the simplicity envelope survives for a geometry-aware code as an inequality (K≤2CA+0.83 for $n=6$ to $12$) that explains under $5%$ of the variance. Results that contradict our hypotheses are reported as results.
No takes yet. Share an insight, caveat, or question.
Luca Eliseo Pavesi (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: