When Structure Doesn't Matter: Loss-Invariant Routing Specialisation in Hybrid Language Models Abstract: We construct a controlled setting in which an architectural channel in a hybrid state-space/attention language model is free to organise its internal representations without measurably affecting the optimisation objective. The channel is "hormone routing": a token-conditional softmax-gated linear combination of a small bank of frozen direction vectors, added to the residual stream after each attention block. We pretrain a 125M-parameter Mamba/MQA hybrid on FineWeb-Edu under a fixed 2B-token budget across five variants of this channel --- extracted directions, random unit vectors, vectors extracted from a randomly initialised model, and a frozen gate at 1.0. The router develops semantically coherent per-genre routing specialisation in every variant, with maximum pairwise L1 distance between genre routing distributions in 0.6--1.6 (uniform = 0, disjoint = 2). The variant of highest specialisation strength is the one whose gate the model cannot modulate. Specialisation strength does not predict downstream loss: the four properly initialised variants converge to identical validation perplexity (26.13). To probe whether this loss-invariance is a ceiling or a regime the model actively absorbs, a sixth variant adds a forced non-zero gate initialisation and a non-trainable 3x amplification. The model drifts the gates downward and shrinks the per-hormone magnitudes, but the residual perturbation it cannot suppress 2x the unforced regime, does not damage loss either. We refer to the regime in which the gated injection is consistently absorbed without affecting predictions as a "homeostatic envelope". The envelope is wider than the unforced variants naturally select; the only axis on which the mechanism can break the model is the training schedule (introducing it during base pretraining damages the loss by +2.07 PPL even though the same router still develops specialisation). The loss-relevant variable is therefore "when" the perturbation is introduced, not "what" it contains nor how strongly it is gated. We additionally describe "intracellular attention", a parameter-efficient slot-attention mechanism inside the SSM scan, as an architectural extension; full empirical evaluation requires a fused CUDA kernel and is reported as preliminary.
A. C. JHA (Mon,) studied this question.