Memorizing Transformers

Catastrophic forgetting remains one of the most persistent structural limitations in neural network design. This paper proposes the Modular Hibernated Memory Architecture (MHMA), a framework that addresses this problem by relocating domain knowledge from a model's active weight space into discrete, hibernated external memory cells activated selectively at inference time. Version 2.1 advances the conceptual foundation of earlier drafts by proposing concrete solutions to five critical engineering challenges: cell construction and format, router accuracy at scale, multi-cell coherence, integration latency, and cell validation and trust. The architecture reframes the catastrophic forgetting problem transforming it from one of weight interference into one of memory management. This paper presents MHMA as a structured conceptual framework, identifies its principal engineering challenges, and establishes a foundation for future prototype development and empirical benchmarking.