Answers are grounded in this paper and cite their sources.
Theoretical analysis demonstrates restoration of empirical process convergence in singular self-attention architectures, revealing geometric mechanisms behind representation collapse.
Modern Transformer architectures undergo severe non-equilibrium phase transitions during training, such as representation condensation and rank collapse. In this paper, we establish Algebraic Geometric Empirical Process (AGEP) theory, a unified mathematical paradigm that characterizes Stochastic Gradient Descent (SGD) learning trajectories on singular schemes. First, we formalize a minimal Softmax-free 2 × 2 self-attention mechanism, defining its rank-1 collapse locus as a closed reduced subscheme Scollapse = Spec(R/ xy-zw ). Applying Hironaka's monoidal blow-up, we prove Theorem 4.1 (Donsker Restoration), demonstrating that the resolution of singularity monomializes the Kullback-Leibler divergence and restores uniform weak convergence (the Donsker property) of the empirical process on exceptional divisor charts, yielding a Real Log Canonical
No takes yet. Share an insight, caveat, or question.
Hideki Ishiyama (2026) studied this question.