Contrastive Projection: Reading Transformer Internals by Differencing Logit Lenses

Subtracting the hidden states of two matched inputs at each layer and projecting throughthe unembedding matrix W_U produces a training-free, layer-by-layer readout of what separatesthem in token space. The subtraction cancels the component the two inputs share and readswhat differs: content that is buried below shared signals in the logit lens of either input alone.We validate the method in two ways: injecting the difference vector at a single layer recoversthe target’s prediction, while an equal-norm, same-frame near-neighbour contrast recoversnothing, and dose-response confirms directional specificity. For compositional contrasts, multi-contrast triangulation recovers token-readable causal content that single-pair readout misses(1% → 77% for “caught a cold”).Applied to Phi-2 (2.7B), the method traces compound-noun recognition to a two-hopMLP→attention circuit, traces both the IOI and factual-recall circuits from their source to-ken to the heads that carry the answer via position-resolved patching, reads what the modelretrieves for real versus fictional entities, reveals that parametric knowledge triggers correctionrather than override when contradicted, traces the divergence between bare and pragmaticallymarked quantifiers, and provides injection evidence that metaphor is per-domain routing ratherthan a figurativity flag, across 15 semantic axes and four models. We release code and data