Exponential Collapse in Softmax Systems: A Scaling Law in Terms of Beta-Delta Invariant (S = beta*delta)

We study collapse in softmax-based exponential normalization and show that it can be described at leading order by a single scalar quantity S_t = β_t Δ_t, where Δ_t is the gap between the two largest scores. We find that the bulk probability mass follows the scaling relation log B_t ≈ −S_t in the regime S_t ≫ log K, and that the collapse threshold scales as S*(K) ~ log K. These results are derived from a decomposition of the log-partition function and supported by numerical experiments across score distributions and perturbations within the softmax framework. The findings provide a minimal and predictive description of collapse as a structural consequence of exponential normalization. Keywords: softmax collapse, scaling laws, exponential normalization, beta delta invariant, bulk probability mass, partition function, log B approx minus S, log B ≈ −S Code & ReproducibilityAll numerical experiments and figures in this paper can be reproduced using the code available at: [shipworm-o/softmax-collapse-figs: Appendix B figures code for softmax collapse phase transition]