Curvature-Corrected Rotary Position Embeddings: An Entropy-Invariant Temperature for Mitigating Numerical-Rank Collapse in Long-Context Attention

As large language models process ever longer contexts, the positional encoding—most commonly rotary position embeddings (RoPEs)—must remain numerically trustworthy. We give an information-geometric analysis of why the attention matrix becomes ill-conditioned as the sequence length L grows. We first show that the tempting frequency-aliasing explanation fails, because RoPE’s multi-frequency code keeps positions well separated. The mechanism is statistical: at fixed softmax temperature the attention entropy grows like logL, and the rows spread their mass over a collision support (inverse participation ratio) that grows polynomially as La (a≈0.9 measured), confining the matrix’s trailing singular values and producing a numerical-rank collapse. We prove, and confirm in 50-digit arithmetic, that the spectral condition number is intrinsically ill-posed here, so the numerical rank is the correct diagnostic. From the exact entropy–temperature identity dH/dβ=−βVarp(e) we derive the entropy-invariant schedule β★(L)=(L/Lref)c0, Curvature-Corrected RoPE (CC-RoPE), and prove that in the diffuse near-circulant regime it eliminates the collapse at a characterised conditioning–mixing cost, whereas no relative-position-preserving warp can. The two signatures are further confirmed in a trained RoPE model (Pythia-160M); the downstream perplexity effect is framed as a falsifiable prediction.