What question did this study set out to answer?

This work examines how routing entropy in Mixture-of-Experts models may be confounded by token position and count, impacting interpretability.

June 23, 2026Open Access

When Routing Entropy Tracks Length, Not Complexity: A Cross-Model Token-Position Confound in MoE Interpretability

Key Points

This work examines how routing entropy in Mixture-of-Experts models may be confounded by token position and count, impacting interpretability.
Analyzed 168 prompts across twelve cognitive complexity levels.
Calculated routing entropy correlations in two Mixture-of-Experts models.
Investigated positional effects on routing entropy across varying prompt lengths.
Routing entropy positively correlates with both cognitive complexity (ρ = +0.80) and prompt token count (ρ = +0.88).
At the final token, the correlation with cognitive complexity becomes negligible (ρ = +0.02).
The observed pattern persists in a separate model run under the same conditions.

Abstract

When Routing Entropy Tracks Length, Not Complexity: A Cross-Model Token-Position Confound in MoE Interpretability. Preprint, version 1. 0 (June 2026). Not peer reviewed. Routing entropy, the Shannon entropy of a mixture-of-experts (MoE) router's per-token weight distribution, is a convenient one-number summary, and a higher value invites reading as a sign that the model is working harder. This note shows that, averaged over prefill tokens, that signal is confounded by prompt token count and token position. Across a 168-prompt suite graded into twelve levels of intended cognitive complexity, the all-token mean routing entropy rises with the intended level in two MoE models (DeepSeek V3. 1, Spearman ρ = +0. 80; Qwen3. 5-397B, ρ = +0. 62), yet it tracks prompt token count at least as strongly (ρ = +0. 88 and +0. 78). The mechanism is positional: later prefill positions carry higher routing entropy because, under causal attention, they read more context, so longer prompts collect more high-entropy late positions and lift the mean. Read at the final prefill token, whose receptive field already spans the whole prompt, the gradient with intended level disappears (ρ = +0. 02 for DeepSeek V3. 1, ρ = −0. 06 for Qwen) and a level-1 versus level-12 comparison reverses sign. The all-token pattern repeats in a DeepSeek R1 run on the same suite. The practical takeaway: routing entropy compared across prompts of different lengths should be reported at a position-controlled readout, or with position and length modeled directly, before a between-prompt difference is read as a difference in what the prompts demand. This deposit contains the paper (LaTeX source and built PDF), the three figures and the script (makefigures. py) that reproduces them and the reported correlations from the source values without re-running any model, a verified bibliography, and a claim-by-claim source index (SOURCES. md). Raw activation/router captures are retained separately and available from the author for verification.

Read Full Paperexternally

AI에게 질문

Bookmark

View Full Paper