HSNNM: A Hebbian Sparse Neural Network Model for Efficient Transformer Learning

We introduce the Hebbian Sparse Neural Network Model (HSNNM), a biologically-inspired trans-former architecture that replaces the standard dense Feed-Forward Network (FFN) with a dynamicallyrouted system of sparse, competing expert sub-networks termed “lobes”. HSNNM integrates three coremechanisms: k-Winners-Take-All (k-WTA) lateral inhibition [4], non-gradient Hebbian plasticity, andnet2net structural neurogenesis. We demonstrate that HSNNM achieves a 49.6% reduction in FLOPsper token (7.18M vs 14.25M FLOPs/token) compared to a matched dense baseline, while incurring a 3.4percentage point accuracy trade-off (76.06% vs 79.45%), a favorable efficiency-accuracy balance giventhe significant compute savings. Through ablation studies, we show that both Hebbian plasticity andk-WTA sparsity contribute to stability over medium-term training (5,000 steps). Our results establishHSNNM as an efficient and robust alternative to dense transformers.