Answers are grounded in this paper and cite their sources.
Demonstrates improved language model performance in continual learning, implying effective expert routing strategies.
Inspired by the role of sleep in biological continual learning, we introduce RVW, a transformer architecture for online continual adaptation of pretrained models. RVW maintains a small pool of per-layer experts that grow and are pruned in response to distribution shift, with no replay buffer and no explicit task identifier. Applied to TinyLlama-1.1B on a 15,000-chunk six-domain stream, RVW reaches 40 average held-out PPL, substantially better than EWC (158), fine-tuning (164), and LoRA (448) on the same parameter-matched base, while preserving prior-domain performance. Threshold sweeps suggest that learned knowledge is encoded in the routing combination of experts across layers rather than individual experts specializing on specific domains.
No takes yet. Share an insight, caveat, or question.
Joshua Ballanco (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: