We present Modus, a novel linear associative memory architecture for sequence modeling that achieves superior performance over both LSTM and Transformer baselines while using a fraction of their parameter budget. Modus replaces the O(L²) self-attention mechanism with a fixed-size O(R²) memory matrix updated via a bounded delta-rule overwrite at every sequence position. On the enwik8 character-level language modeling benchmark, a 156,481-parameter Modus model achieves 3.2713 bits-per-character (BPC), outperforming a 984,832-parameter Transformer (3.5919 BPC) and a 476,672-parameter LSTM (3.4191 BPC)—representing a 6.3× parameter advantage over the Transformer while achieving lower loss. We also demonstrate that Modus exhibits remarkable zero-shot length generalization: trained at sequence length L=128, multi-head configurations maintain 97.7% accuracy on associative recall tasks at L=2048 with no architectural modification. We additionally validate scaling: a 350-million parameter Modus trained for 150,000 steps on the FineWeb-Edu corpus (~7B tokens) on a 16-device TPU Pod achieves a zero-shot HellaSwag accuracy in the ~26-34% range, competitive with GPT-2 Medium (345M params) while using a pure linear associative memory with no attention mechanism whatsoever. Our results constitute a reproducible, empirically grounded demonstration that linear associative memory is a strong alternative to attention, with direct implications for long-context inference efficiency and parameter-efficient pre-training.
Sanyam Chaudhary (Wed,) studied this question.