Noise Accumulation and Rank Collapse in Dense Self-Attention: DSALT

Dense self-attention assigns strictly positive weight to every token in the context window via the softmax, regardless of semantic relevance. This is usually read as the source of attention's expressiveness; we show it is also a structural liability with a precise, measurable signature. A strictly positive row-stochastic matrix has Dobrushin coefficient ρ < 1, so composing dense attention across depth is provably contractive toward a rank-one subspace in the idealized residual-free regime, the mechanism behind the rank collapse of Dong et al. (2021). Because the same coefficient governs how perturbations propagate, contractivity and noise-robustness are two readings of one quantity, and we use ρ as a direct per-layer probe of both. We propose DSALT (Dynamic Sparse Attention with Landmark Tokens), in which each token attends to an adaptive local window plus a small set of landmark tokens selected on the fly by a hybrid-energy score balancing representational distinctiveness against value-projection magnitude. By construction this drives the attention operator away from contractivity (ρ → 1) while cutting per-token cost from O(L²d) to O(L(w̄ + k)d). We train three iso-architecture pairs in which only the attention mechanism differs, plus a graft onto a pretrained Qwen2.5-3B backbone, and the prediction holds cleanly. Dense attention is markedly contractive at depth (mean ρ down to 0.76, individual layers to 0.59), whereas DSALT stays non-contractive at every layer and scale (ρ ≈ 0.98 to 0.9996). Read against two sparse controls, DSALT is the only mechanism that mixes without contracting: sliding-window attention reaches ρ = 1 only by leaving far tokens unmixed, a degeneracy we confirm causally with a counterfactual-context ablation. This sparsity costs under one perplexity point at the training length, yet as context grows the ordering reverses sharply: at 131,072 tokens DSALT beats the dense baseline by 23 perplexity points on the 3B model and by 62 on the deepest one, while running 3.6 to 7.3× faster than a FlashAttention dense baseline. We are explicit about the boundary of these claims. The contraction is measured at the operator level; in trained residual networks the hidden-state effective rank does not collapse at the depths we train, and we report this rather than assert more than the data show. Whether the operator-level contraction reaches the hidden representations at greater depth is a prediction we make and leave open.