Hidden-FFN: Exact CPU Training for Sparse Feed-Forward Networks via Hidden-Neuron Decomposition and Compact Native AdamW

I present a released CPU runtime for exact training of fixed sparse feed-forward networks under the Hidden-FFN(HFFN) decomposition. The key systems rule is simple: support-limited quantities such as sparse weights, sparseweight gradients, and AdamW states stay compact, while activations and activation-space gradients stay dense inpersistent transposed buffers. The runtime executes forward, loss, backward, and AdamW update through nativehidden-major kernels without re-materializing dense masked weights in the hot path.I evaluate 213 one-epoch CPU configurations across multiple FFN sizes, densities from 5% to 100%, andbatch sizes {8, 32, 64} on the same Apple M1 4-core platform used in the other HFFN papers. HFFN is stronglyfavorable at low density: at 5%, median end-to-end speedups over a masked dense PyTorch CPU baseline are7.03×, 4.74×, and 3.17× for batch sizes 8, 32, and 64; at 20%, they remain 3.23×, 2.13×, and 1.35×. The regimebecomes transitional around 30% and unfavorable at high density and large batch. Memory improves moredirectly than time, with median dense-reference/HFFN total-memory ratios of 7.15× at 5% density and 4.83× at10%. Short multi-epoch checks confirm that the native runtime follows the masked dense training trajectory.