What does this research mean for the field?

Learning success in neural systems is governed by an effective capacity constraint that predicts sharp phase transitions in test accuracy based on the scaling of Shannon entropy, model capacity, and dataset size. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

This research aims to understand why neural learning systems fail on tasks that appear feasible, focusing on entropy, capacity, and scaling.

May 16, 2026Open Access

A Constructibility Bound in Neural Learning Systems: Phase Transitions in Entropy–Capacity–Data Scaling

Key Points

This research aims to understand why neural learning systems fail on tasks that appear feasible, focusing on entropy, capacity, and scaling.
Investigated the relationship between Shannon entropy, model capacity, and dataset scale in neural learning systems.
Utilized transformer architectures (DistilBERT, BERT-base, RoBERTa-large) and benchmarks (IMDb, SST-2) for analysis.
Developed the Constructibility Framework, which includes empirical scaling laws and theoretical lemmas.
Observed sharp transitions in test accuracy as datasets scaled, with curve-collapse coefficient λ = 0.331 derived.
Empirical scaling law E(S) ~ H1.42 / (C0.31 · n0.47) attained R² = 0.91 across architectures and benchmarks.

Abstract

We study the relationship between Shannon entropy, model capacity, and dataset scale in neural learning systems, asking: why do learning systems abruptly fail on tasks that remain structurally feasible? We propose the Constructibility Framework, in which learning success is governed by an effective capacity constraint L(S) = Cβ · nγ / Hα. Through controlled entropy injection and systematic scaling of transformer architectures (DistilBERT, BERT-base, RoBERTa-large) on two benchmarks (IMDb, SST-2), we observe sharp, reproducible transitions in test accuracy. We resolve a structural gap in prior formulations via Lemma 1 (Risk Bridge Lemma), introduce Theorems 4 and 5, and verify Assumption A3 analytically via Proposition 1. The curve-collapse coefficient λ = γ/α = 0.331 is derived from first principles. An empirical scaling law E(S) ~ H1.42 / (C0.31 · n0.47) is fit with R² = 0.91 across architectures and both benchmarks.

A Constructibility Bound in Neural Learning Systems: Phase Transitions in Entropy–Capacity–Data Scaling

Key Points

Abstract

Cite This Study