Comparative analysis reveals distinct expert specialization during token generation in mixture-of-experts models, indicating prior negative findings stem from prefill-only evaluation.
Generation-Time Routing Reveals Expert Specialization That Prefill Measurements Miss. Preprint, version 1.1 (August 2026; version 1.0 June 2026). Not peer reviewed. In two Qwen3.5 mixture-of-experts (MoE) checkpoints (35B and 122B parameters) run on 60 prompts spanning twenty academic domains, the expert that receives the most routing weight in a domain depends on which regime of the forward pass is measured. Over prefill, when the prompt is processed in parallel, one generalist expert wins 18 of 20 domains in the smaller model and 13 of 20 in the larger, and routing looks domain-blind. Over generation, when the answer is produced token by token, the winners disperse to 20 and 18 distinct experts and the generalist wins one or two. The dispersion is a property of the regime: with prefill and generation matched in token count, the two regimes select nearly disjoint top-expert sets (Jaccard 0.00 to 0.23) at unchanged per-token routing entropy (0.958 versus 0.953), so generation spreads each token over a similar number of experts and routes it to different ones, and the pattern reproduces across model sizes with unrelated expert indices. The widely cited negative result that MoE experts do not specialize by topic was measured over supplied text, which is prefill routing. Expert-specialization measurements over supplied text, or pooled across regimes, are therefore understated by construction; prefill-based negative results are silent about the regime in which specialization is expressed; and analyses that ask what experts specialize in should separate the two regimes and read specialization from generation. Changes in this version. Single-thesis revision (August 2026): one claim carried abstract to conclusion, token-balanced control foregrounded, proposed-experiment lists removed, terms defined at first use. All version 1.0 numeric values retained. This deposit contains the paper (LaTeX source and built PDF), a claim-by-claim source index (SOURCES.md), and the supporting data directory, unchanged from version 1.0.
No takes yet. Share an insight, caveat, or question.
Jeffrey W. Shorthill (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: