What question did this study set out to answer?

To present a novel recurrent architecture for language modeling and classification that optimizes memory usage.

April 30, 2026Open Access

Echo-DSRN-114M: Surprise-Gated Dual-State Recurrent Architecture for Efficient Language Modeling and Classification

Key Points

To present a novel recurrent architecture for language modeling and classification that optimizes memory usage.
Developed Echo-DSRN-114M hybrid recurrent architecture with GRU fast state and surprise-gated slow memory state.
Implemented bounded sliding-window attention maintaining O(1) memory.
Used a prototype version, reporting early outcomes on structured retrieval tasks.
Outperformed Pythia-160M in zero-shot structured retrieval with SciQ scoring 0.583 compared to 0.519.
Maintained efficient memory management with bounded attention cache fixed at 128 tokens.

Abstract

Echo-DSRN-114M is a hybrid recurrent architecture combining a GRU fast state, a surprise-gated slow memory state, and bounded sliding-window attention, maintaining O (1) memory with a bounded O (windowₛize) attention cache fixed at 128 tokens per layer. The surprise gate uses per-token prediction error to selectively write to long-term memory, outperforming Pythia-160M on zero-shot structured retrieval (SciQ: 0. 583 vs 0. 519) at 114M parameters and 219 MB fp16. Weights, training code, LoRA adapters, and live telemetry applications are publicly available under Apache 2. 0 in the Echo-DSRN HuggingFace collection. This is a preliminary working paper reporting results from an undertrained prototype. Findings are provisional and have not been peer-reviewed.

Read Full Paperexternally

Demander à l'IA

Bookmark

View Full Paper