Maestro: Learning-Based Reuse of Activated Experts to Mitigate Long-Tail Latency under Heavy MoE Workloads in Cloud

Cloud LLM services increasingly adopt Mixture-of-Experts (MoE) models for cost-effective scaling, yet under heavy multi-task workloads their serving performance is often bottlenecked by severe tail latency. The execution semantics of MoE inference are poorly aligned with cloud-native serving abstractions, causing spatial and temporal mismatch in expert placement and demand, and consequently frequent expert loading. Expert demand, however, exhibits structured locality where correlated requests and decoding steps often induce overlapping expert working sets, creating opportunities for activated-expert reuse. Exploiting such reuse is non-trivial because request routing, expert reuse, and resource allocation are tightly coupled over time, forming a sequential decision problem whose coupled dimensions existing systems address only in isolation. This paper presents Maestro, a learning-based framework for activated-expert reuse in cloud-native MoE inference. Maestro models tasks, experts, instances, and execution stages as a heterogeneous graph with relational message passing to capture structured system state, and combines a lightweight Neural Optimistic Value Policy (NOVP) with Pareto-guided action ranking (AdaFilter) for efficient online decision making. It further introduces a prefill-driven expert prefetch mechanism that overlaps loading with computation, where the per-window transfer budget is jointly determined by the RL controller to balance reuse coverage against memory pressure. Evaluated on Mixtral-8 × 7B, Phi-3.5-MoE, and Qwen1.5-MoE, Maestro reduces P50 and P99 latency by up to 34.02% and 28.94%, respectively, and achieves up to 1.51 × the throughput of vLLM. Against the SOTA learning-based methods, it reduces P99 and TTFT by 8.58%–17.47% and 6.22%–17.43%, respectively, while improving throughput by 12.09%–22.70%.