The Gauge Freedom of Attention-Head Evidence: Measuring and Materializing Latent Head Competition in Transformers

Multi-head attention normalizes each head independently, so the log-sum-exp (LSE) of ahead’s logits, a natural “evidence” score that standard kernels already compute, never affectsthe model’s output. We show it is also not directly interpretable. Softmax is invariant to per-row additive shifts, so the LSE carries a gauge component that training never constrains, andin public checkpoints this component can dwarf the signal: in Pythia-410M, offsets of order 104explain essentially all cross-head LSE variance. We introduce a gauge-fixed probe, the centeredLSE, and find that raw LSE misleads in both directions. It understates concentration in GPT-2 (4.8 vs. 2.5 effective heads of 12) and overstates it in Pythia-160M (1.1 vs. 1.6), with thedirection consistent within each of the five model families we probe. Gauge-fixed, all twelvecheckpoints (70M to 2.8B parameters) show concentrated head evidence, between 13% and 55%of heads effective, and every one contains layers whose evidence a single head dominates. Thisconcentration is largely unrelated to attention sinks. We then materialize the latent competition:normalizing jointly over heads and keys factorizes exactly into per-head attention times an LSE-derived head gate, giving a parameter-free mechanism (Evidence-Gated Attention) with vanillaattention as its τ →∞limit. Used as an intervention, it pins the gauge and trains without headcollapse, and its gradient, not its forward gating, carries the effect. What the intervention doesto the model depends on how long training runs. At 49M tokens it costs +0.031 validation natsand raises separator-anchored sink mass by 1.55×; at 491M tokens it wins by 0.009 nats (2.9σ,non-overlapping seeds) and shows roughly a third of the baseline’s sink mass, with the samedirections holding at 124M parameters. Both the cost and the sink effect of head competition arebudget-dependent, and the gauge is a confound for any LSE-based head-importance analysis.