Answers are grounded in this paper and cite their sources.
Dynamical systems analysis reveals value matrices govern attractor selection and token clustering in reduced transformers, indicating value representations drive forward-pass consensus.
We study an interacting particle system on the unit sphere Sd−1 motivated by the forward-pass dynamics of a transformer model that combines a value matrix V and a u-aligned multi-layer perceptron (MLP) Φ, a particular form of MLP specified below. For a symmetric value matrix and this MLP, the one-cluster dynamic is the Riemannian gradient flow of an explicit Lyapunov function L=fV+g, which combines the Rayleigh quotient of V with the MLP potential. Every trajectory converges to a fixed point, limit cycles are excluded, and the fixed points are the critical points of L: local maxima are stable attractors and saddles are unstable. Isolating the value matrix (Φ≡0, with V possibly non-symmetric and a real strictly dominant eigenvalue λ1), we show that the dominant eigenvector v1 is the selected attractor and that its basin is almost the whole sphere, the two basins of ±v1 being separated by an invariant subsphere carried by the left eigenvector of V. The subdominant eigenvectors form a hierarchical structure of saddles, and a complex dominant pair replaces the point attractor by a great-circle limit cycle. When the tokens move independently, softmax attention contributes no first-order clustering force: the tokens collapse to ±v1 at a consensus rate equal to λ1 and independent of the attention matrix, and whether two clusters in opposite basins coexist or merge is governed by the single order parameter s=v1⊤Av1. Thus, the value matrix, rather than the attention, controls the location and the selection of the limiting cluster. All results are confirmed by numerical integration of the continuous-time ordinary differential equation (ODE).
No takes yet. Share an insight, caveat, or question.
Vitaly Volpert (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: