Component–Optimizer Assignment in Transformers Is Binary and Static: Negative Results for Learned Routing and a Positive Result for Embedding Orthogonalization

Muon, an optimizer that spectrally normalizes the momentum-accumulated gradient of two-dimensional weight matrices, is conventionally applied only to hidden-layer matrices, with embeddings, the language-modelling head and one-dimensional parameters left to AdamW. This split is a hand-chosen convention. We ask whether it can instead be learned, per component and per training step, from measurable properties of the gradient. We build DORoT (Dynamic Optimizer Routing for Transformers), a system that measures per-component gradient geometry, converts it into a routing prior, and refines that prior with a bilevel meta-objective that differentiates a one-step lookahead validation loss through soft routing weights. On a 29.94M-parameter GPT-style model trained on FineWeb-Edu, we report a negative result along four independent axes: learned bilevel routing is never better than a frozen prior; a geometry-differentiated prior loses to a uniform one; the blend level spans only 0.024 validation loss across the range [0.70, 0.98]; and time-varying schedules lose to constants by 0.060–0.151. A diagnostic we introduce explains the first failure mechanistically: the one-step objective pushes away from Muon on 75–100% of outer steps for every routable component, because a single-step lookahead cannot observe a conditioning benefit that accrues over many steps. The same experimental apparatus yields a positive finding: extending Muon to the token embedding matrix — excluded by convention, yet 19.4M of the model's 30.0M parameters — improves validation loss by 0.157, and after re-tuning the Muon learning rate and weight decay the resulting purely static configuration reaches 4.858 versus 5.516 for AdamW at matched budget and seed, a 48.2% reduction in perplexity. We conclude that at this scale component–optimizer assignment is a large but binary and static effect, and that the continuous, dynamic dimension DORoT was designed to exploit is essentially flat.