Hidden-FFN: Exact Inference for Sparse Feed-Forward Networks via Hidden-Neuron Decomposition

Feed-forward networks (FFNs) are a major source of parameters and inference cost in modern neural architectures.While pruning can produce sparse FFNs, practical speedups remain difficult because generic sparse formatsintroduce index reads, indirect memory accesses, and scatter overheads. I introduce Hidden-FFN (HFFN), an exactsparse FFN inference method that rewrites the FFN as a sum of independent hidden-neuron contributions. Eachhidden neuron is represented by a sparse up-projection support, an exact GELU activation, and a sparse down-projection support. This decomposition exposes structurally simple task-level parallelism, because hidden-neuroncontributions are independent until the final output reduction.I evaluate a compact HFFN format and a specialized Apple M1 CPU kernel on synthetic sparse FFNs withcontrolled density. The primary baseline is dense ONNX Runtime CPU inference on the equivalent dense FFN,while PyTorch sparse is included as a generic sparse framework baseline. I also include a controlled two-stage sparseC++ baseline to separate HFFN from PyTorch sparse framework overheads. On the tested 512 → 2048 → 512sweep, HFFN reaches geometric speedups of 3.49×, 2.32×, and 1.44× over ONNX at 5 %, 10 %, and 20 % density,while reducing storage by 92.0 %, 84.5 %, and 69.5 %. The advantage disappears at high density, where sparse-indexand indirect-access costs dominate.