HFFN-Transformer: Integrating Exact Sparse Feed-Forward Kernels into Transformer Inference

Feed-forward networks (FFNs) dominate a large share of Transformer parameters and inference work. Hidden-FFN (HFFN) was previously introduced as an exact sparse FFN method based on hidden-neuron decompositionand validated on isolated FFN blocks. This paper studies the next question: does HFFN remain useful insidecomplete toy Transformer forward inference?I introduce an evaluation runtime that replaces Transformer FFN sublayers with HFFN while keeping attention,embeddings, normalization, residuals, and the language-model head in the surrounding PyTorch graph. Thereported benchmark path replaces FFN sublayers only; the tighter fused post-attention entry point and full-graphLLM runtime support are left for future work. The system is evaluated end-to-end on a controlled family of 17toy Transformer models covering 3 dense models, 14 sparse models, and 56 sparse model/sequence configurationsover sequence lengths 1, 8, 16, and 32 on an Apple M1 CPU setup.