Lex-Δ: Post-hoc Replacement of Pretrained Transformer FFNs by Decomposing Them into Token Lookup, Linear Response, and Residual Nonlinearity

Feed-forward networks (FFNs) hold roughly half of the parameters of a dense language model and act as key-value memories. Prior work either adds token-indexed memories next to the FFN, or trains a lookup-style FFN from scratch. We instead study how far an FFN that has already been pretrained can be replaced after the fact. We decompose each pretrained FFN's output into (0) a token-dependent mean, (1) a context-proportional linear response, and (2) a residual nonlinearity, obtain the first two terms in closed form from teacher statistics (fixed-effects ridge regression, reduced-rank truncation, frequency-weighted SVD into a code shared across layers, and a dedicated position-0 vector), and learn only a small residual block, followed by light distillation. On LFM2.5-230M, held-out R^2 of the FFN output is 0.38 for the token mean, 0.66 for a single linear map, and 0.73 for both; the split varies strongly by layer (0.80 token-mean at layer 0, 0.91 linear at layer 5). Replacing all 14 FFNs with Lex-Delta using 17.5% of the FFN parameters on the GPU (19.3M of 110.1M) and 49M tokens of distillation gives a mean KL to the teacher of 0.567 across seven held-out domains, versus 0.479 for a pruning-and-distillation baseline that keeps 50% of the FFN. On SciQ the replaced model keeps 84-88% of the teacher's above-chance accuracy and chooses the same answer as the teacher about 80% of the time. We also report what did not help: bigram lookup tables, lookup+linear without a nonlinear residual, and layer-wise distillation on student inputs. This is a single-model, small-scale preliminary study.