Evolving Sparse Spiking Mixture-of-Experts: A Unified Neuromorphic Language Modeling Framework

Modern large language model deployment is constrained by severe economic, metabolic, and utility-driven bottlenecks. We present the Evolving Sparse Spiking Mixture-of-Experts (S²-MoE), a unified neuromorphic framework targeting the core economics of running LLMs in production, with a two-pronged value proposition: (1) a Distilled Deployment Moat, extracting dark knowledge from existing dense models so a sparse spiking student matches its dense teacher's quality while cutting active FFN compute by over 80%; and (2) a Datacenter Energy Disruption, delivering a 3.0× to 7.1× reduction in active GPU energy consumption. S²-MoE synthesizes event-driven representation via Leaky Integrate-and-Fire (LIF) neurons, sub-quadratic sequence modeling via a matrix-state Gated Linear Attention (GLA) backbone, and a structurally dynamic routing network that evolves via simulated offline sleep-consolidation phases. Exposing a "Protocol Trap" under warmup+cosine scheduling where standard sparse experts starve, we introduce an always-on shared expert topology and peer self-distillation. On an NVML-extended energy harness (H100 and commodity RTX GPUs), the distilled shared-core student is quality-competitive with its dense teacher across corpora (−5.15% perplexity on FineWeb-Edu, −4.4% on TinyStories at Ne=32, 3 seeds), while cutting active FFN compute by 95.3% (21.5× efficiency at Ne=32, 43× at Ne=64) and reducing inference energy by 3.1–4.1× (Ne=32) to 5.1–7.1× (Ne=64) on H100. We are precise about the scope of the quality win: under 4× extended training the crossover is corpus-dependent — on hard text it reverses (dense overtakes, +2.4% ± 0.8%), while on structured text it survives (−2.6% ± 0.3%). The durable, scale- and budget-invariant contribution is the shape-determined efficiency win. We further show the Top-1 data-diversity starvation ceiling is structural: six optimization-, routing-, depth-, and capacity-level interventions were built and rejected, leaving the shared core and peer self-distillation as the only effective quality levers. The joint win scales wedge-free to Ne=64 via a PyTorch compaction fix.