A Visibility Threshold for Top-k Logit-Lens Readouts

Projecting a transformer hidden state, or a difference of hidden states, through the unembeddingmatrix W_U and reading the top-k tokens is a standard inspection technique. We ask how much of thestate’s energy must be aligned with a token direction for that token to appear in the top k. Under anisotropy assumption on the remainder and on the rows of W_U , the top-k readout is a max-statistic andthe answer is a closed form: f ∗ = 2 ln(2V )/(d+2 ln(2V )), about 0.9% of the state’s energy for Phi-2(d=2560, V =51,200). We test this with planted signals on ten models from five families (Pythia, Phi,GPT-2, BLOOM, Qwen, TinyLlama; d = 512–2560, V = 32k–251k). On the eight models whoseunembedding passes the isotropy gate, the measured half-visibility point f50 tracks f ∗ with slope1.15 in the log–log regression (R2 = 0.98), with a family-dependent constant f50/f ∗ ∈ [0.56, 0.70](0.56 ± 0.003 within the Pythia/Phi subset); the ceiling’s predicted rise with ln V is directionallyconfirmed across the 8× vocabulary span. The two smallest Pythia models violate the assumption:their unembedding row directions are heavy-tailed (bath excess kurtosis +1.1 and +2.2 versus ≈ 0),the empirical ceiling is about twice the isotropic prediction, and the threshold rises accordingly (forPythia-70m, a planted signal carrying 30% of the state’s energy is usually invisible). Averaging Nindependent difference vectors that share a signal lowers the threshold roughly N -fold, as predicted.Real hidden states violate the isotropy of the remainder as well (post-signal kurtosis 2.8–19.5), raisingthe effective bar for real reads by 1.05–1.5× on the gate-passing models. This is a measured correction,not a failure of the framework. A battery of eight real contrast pairs clears the bar at 4.5–16× onevery isotropic model (24/24 present, surviving the real-remainder correction) and is absent on bothanisotropic models (16/16). The practical readings are: a clean top-k readout is compatible with thelarge majority of the state’s energy being unaligned with any token direction; absence from the top kbounds aligned energy only up to f ∗; and the threshold is a property of W_U geometry, not of d alone,so small-model readouts require a measured calibration.