Matrix Context: Mixture-of-Contexts RAG for Robust and Inspectable Agent Memory

Matrix Context: Mixture-of-Contexts RAG for Robust and Inspectable Agent Memory We introduce Mixture-of-Contexts RAG (MoC-RAG), a routed retrieval architecture for agent memory that selects a small set of typed context experts before retrieving, then assembles a token-budgeted, inspectable context pack. To test the architecture beyond a keyword-aligned fixture, we build a public benchmark of typed agent memory (1,000 context items, 600 queries, eight experts, six domains) in which every gold fact is surrounded by five kinds of hard negative and queried in five styles ranging from direct to adversarial, with the same test topics phrased into parallel keyword, paraphrased, and adversarial splits. On this benchmark MoC-RAG improves robustness over flat RAG baselines: under adversarial lexical shift a strong lexical baseline (BM25) drops from 100% to 64% Recall@8 (−36 points), whereas MoC-RAG holds within ~17 points and overtakes BM25 on the adversarial split by ~15 points, while carrying roughly half the hard distractors of dense, hybrid, metadata-filtered, and reranked RAG at 95–100% expert-routing accuracy. These results do not show that MoC-RAG is universally better than RAG; they show a narrower but important advantage: when agent memory is typed, noisy, and queried under paraphrase or misleading wording, routing before retrieval yields cleaner and more robust context.