Feature Transport Across Grammatical Roles in Pythia-410M

Cross-position information flow in transformers has been studied through subject–verbagreement [Hao and Linzen, 2023], pronoun resolution circuits [Wang et al., 2023], bindingsubspaces [Feng and Steinhardt, 2023], OV singular-vector analysis [Ahmad et al., 2025],and activation transport operators [Szablewski and Masiak, 2025]. We measure one specificmechanism—a low-rank linear map from a source token’s representation to a consumingposition—across three grammatical roles in Pythia-410M-deduped. Object channels (cen-tered R2 0.52–0.56) are stronger than the subject channel (0.41), despite English having noobject agreement. Standard grammatical features (number, gender, animacy) account for27% of the subject channel’s variance; unsupervised clustering of the remaining contentrecovers 7 semantic axes (place, body-part, food, person, animal, household object, waterbody), each of which causally shifts next-token predictions when patched, while matchedrandom-direction controls do not. The transported copy is decodable at the consuming posi-tion from layer 8 but only becomes causal at layers 20–23. All results are from one model andone language (English). Code and data: https://github.com/EvidentSolutions/Deposit.