MoOE (mixture of optimized experts) : Bounded Expert Streaming and Neural Orchestration for Sub-Latency Sparse MoE Inference on Consumer Hardware

Executing high-parameter mixture-of-experts (MoE) architectures on asymmetric, consumergrade hardware is often bottlenecked by weight-transfer latency across the PCI Express (PCIe)bus. We present MooE, a routing and memory-management framework designed to reduceor eliminate transfer-induced stalls during inference [1]. The design combines a hard ≤ 1 GBweight-size bound for each expert, early semantic routing, asynchronous prefetch, and a dedicated 10-million-parameter neural orchestrator. Under the assumed PCIe 4.0 ×16 bandwidthand sufficient independent computation, the bounded transfer can be overlapped with severaltransformer layers. In this way, consumer GPU memory is treated as a dynamic, predictivelymanaged ring buffer rather than as storage for the complete model. We describe the proposedexecution pipeline, the orchestrator state and actions, the SSD-to-RAM-to-VRAM storage hierarchy, and the assumptions required for latency masking.