What question did this study set out to answer?

This research aims to explore a fast-weight memory mechanism for continual learning without backward updates during inference.

July 8, 2026Open Access

A Trained Fast-Weight Memory: Continual Rule Binding at Inference Without Backward

Key Points

This research aims to explore a fast-weight memory mechanism for continual learning without backward updates during inference.
Utilized a 3M-parameter transformer with an 8-slot memory bank on a modular rule task.
Tested the model's ability to install, retain, and replace rules based on forward passes without traditional training methods.
Implemented a curriculum to train the read/write memory circuit using diverse conversation structures.
The model achieved held-rule accuracy of 0.79-1.00 from a single presentation.
Post-switch accuracy was 0.95 during training and 0.78 for held rules; old-rule persistence dropped to 0.000.
TTT showed a 138× higher cost per rule update with no benefit for unseen queries.

Abstract

Continual learning at inference usually means test-time training (TTT): gradient steps on a clone of the model. We study the alternative the fast-weight literature has long promised: a small bank of vectors, written by the forward pass itself, that modulates subsequent computation — no backward, no optimizer, no weight copy. On a keyed multi-turn rule task (a fresh modular rule per conversation, unseen queries, K=2 concurrent rules), a 3M-parameter transformer with an 8-slot bank learns to (i) install a never-trained rule from a single 13-token presentation (held-rule accuracy 0.79–1.00 across two seeds; chance 0.008), (ii) retain it across turns and slot eviction, and (iii) replace it mid-conversation in one forward pass (post-switch accuracy 0.95 train / 0.78 held; old-rule persistence 0.000). Head-to-head on the same conversations, TTT with a full learning-rate sweep memorizes its adaptation examples (pair accuracy 0.99) yet transfers exactly nothing to unseen queries, while costing 138× more per rule update and destroying 62% of the untouched concurrent rule; in-window ICL is also at chance — the bank is the model's only functional adaptation pathway. Crucially, none of this is emergent from the architecture: the same architecture trained without mid-conversation rule switches perseverates completely (old-rule persistence 1.000 zero-shot). Memory policy — what to keep, when to overwrite, how to write on a dirty bank — is a trained behaviour, installed by randomizing conversation structure at training time. We map the boundary (an untrained rule family defeats bank and TTT equally: the limit is the meta-learned envelope, not the mechanism) and report a seed-level bifurcation between selective-update and flush-and-rewrite replacement policies. Training the read/write circuit requires breaking an ignore-the-bank fixed point; we give the recipe (teacher-forced code bootstrap with annealed blending, mastery-gated rule curriculum) and the diversity threshold below which the read memorizes instead of generalizing. Version of record of the repository master at tag v0.2-preprint

Read Full Paperexternally

AI에게 질문

Bookmark

View Full Paper