Allocation for Efficient Sparse Language Model Rou

text Large language models (LLMs) typically allocate computation uniformly across tokens, ignoring substantial variation in token-level predictive uncertainty. We propose Entropy-Guided Routing (EGR), a mechanism that dynamically allocates computation based on the predictive entropy of each token. Unlike learned gating networks in sparse mixture-of-experts (MoE) models, EGR uses entropy as a lightweight, interpretable, and zero-parameter routing signal. We provide an information-theoretic justification, connecting EGR to the Information Bottleneck principle. Empirically, EGR achieves improved compute-efficiency Pareto frontiers compared to dense transformers and Switch Transformer. On WikiText-103, EGR attains a perplexity of 17.8 using only 55% of the dense model's FLOPs, outperforming Switch Transformer (18.9 PPL, 65% FLOPs). Ablation studies demonstrate that entropy-based routing matches learned gating performance while introducing no additional parameters. Our results establish entropy-guided computation as a principled and practical direction for efficient language modeling. .