What question did this study set out to answer?

The aim is to develop an attention-free architecture for efficient language modeling while maintaining constant memory usage.

May 21, 2026Open Access

Hierarchical Gated Delta Memory: Attention-Free Language Modeling with Constant Inference-Time Memory

Key Points

The aim is to develop an attention-free architecture for efficient language modeling while maintaining constant memory usage.
Developed an architecture called Hierarchical Gated Delta Memory for byte-level language modeling.
Implemented techniques like gated outer-product dynamics and multi-timescale decay.
Evaluated the method through training and experiments on language modeling and long-context tasks.
HGDM demonstrated efficient long-context inference while maintaining fixed memory usage.
Achieved competitive performance in byte-level language modeling tasks compared to existing models.
No self-attention or external key-value caches needed, leading to reduced complexity.

Abstract

Hierarchical Gated Delta Memory (HGDM) is an attention-free recurrent architecture for byte-level language modeling with constant inference-time memory. HGDM maintains a fixed-size recurrent matrix state updated through gated outer-product dynamics and multi-timescale decay mechanisms inspired by predictive coding and sparse representations. The architecture eliminates self-attention and external key-value caches while supporting long-context recurrent inference with fixed memory usage. This release contains the accompanying preprint describing the HGDM architecture, training methodology, Nitro Triton scan kernel, and experimental evaluations on byte-level language modeling and long-context retrieval tasks. Code repository:https://github.com/iam-saiteja/HGDM-Hierarchical-Gated-Delta-Memory

Hierarchical Gated Delta Memory: Attention-Free Language Modeling with Constant Inference-Time Memory

Key Points

Abstract

Cite This Study