Transfer Grokking: Linear Projection of Fourier Representations Across Model Residual Streams

Grokked transformers learn perfectly generalizing modular arithmetic circuits thatencode discrete Fourier features in their residual stream. Transferring these com-piled representations into large language models (LLMs) would enable neuralreuse of structured knowledge, but prior attempts have found two independentbarriers: (1) mean-squared-error (MSE) projection training produces weight ma-trices that are perfectly probeable yet misaligned with the target LLM’s unem-bedding directions, and (2) injecting the projected representation at intermediatelayers corrupts the LLM’s remaining computation. We show that both barriersare resolvable. Training the projection via cross-entropy through a frozen unem-bedding layer achieves perfect logit-lens accuracy (1.0), while injecting at thefnal layer (L31 for Phi-2) achieves perfect task accuracy (1.0) at full strength(α = 1.0), while intermediate injection strengths (α ≤ 0.7) leave both task ac-curacy and language-modeling perplexity statistically indistinguishable from Phi-2’s unpatched baseline. This discontinuity—a hard override rather than a smoothtradeoff—is exact across all five seeds and both operations, and follows from alayernorm-induced norm mismatch between the projected state and the residualstream. Results replicate across modular addition and multiplication(both mod 97), demonstrating that the transferred representation is task-agnostic.Our fndings suggest that grokked and language-model representations are fun-damentally commensurable - the bottleneck is not geometric but operational, andresolvable by targeting the last layer.