Answers are grounded in this paper and cite their sources.
Autoregressive Large Language Model (LLM) decoding at low batch sizes is fundamentally bottlenecked by memory bandwidth, with gated Feed-Forward Networks (SwiGLU / GeGLU) accounting for 85–90% of parameter traffic in modern Grouped-Query Attention (GQA) Transformers. Existing sparse inference engines face a structural trilemma: learned-predictor runtimes (e.g., DejaVu, PowerInfer) require ReLU conversion and auxiliary predictor training that inflates VRAM; gate-thresholding methods (e.g., CATS) still read 100% of the dense gate projection; and coordinate-pruning methods (e.g., TEAL) corrupt exact inner products past 40–50% sparsity. We introduce FFMod (v0.2.0), a zero-training, zero-calibration sparse LLM inference architecture in Triton and PyTorch built on separating static knowledge storage from dynamic input-dependent compute planning: Self-Indexed Dual-Branch Smooth-INT4 x Top-D Router: Constructed analytically from any pretrained SwiGLU or GeGLU weight triplet (Llama, Qwen2/2.5/3, Mistral, Gemma-2, Phi-3, DeepSeek) with zero calibration data, scanning <5% of dense FFN memory traffic to identify winning channels and evaluating them across the full hidden dimension H (compatible with torch.compile(fullgraph=True), CUDAGraph, and speculative decoding micro-batches N=1..8). Negative-Overhead BitSplit-12 and FP8 Sparse Kernels: Eliminates the index memory overhead while accelerating execution. On an NVIDIA GeForce RTX 5090 Laptop GPU (Blackwell, 24 GB GDDR7), Qwen2.5-3B VRAM drops by 13.0% in BitSplit-12 (5.37 GB vs. 6.17 GB dense BF16) and by 26.1% in FP8 (4.56 GB), achieving 2.36x–3.54x cold-cache FFN speedups, 1.51x speedup over dense cuBLAS Tensor Cores at speculative N=4, and 1.57x–1.70x end-to-end decoding throughput (91.6 to 143.6–155.2 tok/s). KV-Safe Hybrid Layer Scheduling & 12-SM Open-Page DDR5 PCIe 5.0 Tiered Engine: Uncovers the KV-Cache Pollution Paradox across transformer depth and co-designs a 12-SM coalesced UVA gather kernel (38.83 GB/s random channel gather over PCIe 5.0 x16, +30.2% over 82-SM bank-thrashing UVA while leaving 85.4% of GPU SMs free for 96–99% compute/DMA overlap). By prefetching a high-confidence 35% channel core prior to Self-Attention (92.6% precision) and overlapping exact post-attention delta fixup inside a single two-stream fork-join CUDAGraph, FFMod runs 64-layer Qwen2.5-32B-Instruct (65.53 GB BF16) with 100.0% Top-1 agreement (KL = 0.0139) at 3.84–5.40 tok/s (4.74x–6.67x speedup) in 21.1 GB VRAM and 3.41–4.29 tok/s (4.32x–5.43x speedup) under a strict 15.6 GB VRAM cap. Unified VRAM-First Auto Allocator & Persistent Sharded Cache (.ffmod_cache_v1): Uses 0 GB host RAM whenever a model fits in GPU VRAM, and persists canonical budget-independent Smooth-INT4 + FP8 shards (1.81x smaller on disk, hydrating 64 layers in ~10.8 s from PCIe NVMe with zero re-quantization), exposed via an OpenAI-compatible HTTP/SSE server (ffmod-serve) and a vLLM plugin. Open-Source Code & Reproducibility Suite: https://github.com/ferencfresz/ffmod01
No takes yet. Share an insight, caveat, or question.
Ferenc Frész (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: