Experimental evaluation demonstrates efficient browser-native MoE inference via WebGPU and expert paging on consumer GPUs, indicating practical local execution of large quantized models.
webgguf runs quantized GGUF language models in a browser tab through WebGPU, without model conversion or server-side inference. Its main mechanism is a fixed-size expert arena: routed MoE weights stay resident when their working set fits the VRAM budget and are paged from OPFS when it does not. This allows a 35B hybrid MoE to run on a laptop GPU with 16 GiB of VRAM while retaining space for the KV cache, recurrent state, activations, and browser allocations. On an RTX 4090 Laptop GPU at a declared 150 W power limit, the best observed warm turn on Qwen3.6-35B-A3B Q2_K reaches 77.0 tok/s. Under the replicated 10-turn protocol, the median steady-state step rate is 71.6 tok/s and the wall-clock generation rate is 60.2 tok/s. Dense decode reaches 224.1 tok/s on Qwen3.5-0.8B Q4_0. In a separate same-host fresh-context comparison on that model, webgguf decodes 2.26 times faster than WebLLM 0.2.84; the TTFT ranges overlap. The evaluation checks individual kernels against an f64 CPU reference, model-level agreement against llama.cpp, shader enablement, and end-to-end performance. The agreement test was also replayed on an Apple M4 Pro; cross-host disagreements occur only at low-margin oracle positions. Conversation checkpoints include both the attention KV cache and the recurrent Gated DeltaNet state. Measurements, runners, and reference outputs are distributed with the implementation.
No takes yet. Share an insight, caveat, or question.
Cristiano Mazzella (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: