When Benchmarks Stop Testing Memory: Protocol Isolation Collapse in Long-Context Retrieval Evaluation

We demonstrate that LongMemEval (ICLR 2025), the de facto standard for evaluating AI agent memory, does not test memory retrieval. Its per-question isolation protocol pre-selects ~50 candidate sessions per question, reducing the task to keyword matching. The same FTS5/BM25 system achieves 94.0% R@5 under the standard protocol but collapses to 34.8% when searching globally across all 19,869 sessions—a 59.2 percentage point gap from a single protocol change. Typed-λ graph traversal yields zero gain under the standard protocol (McNemar p=1.00). We provide reproduction code, raw results, and an archived snapshot of the official LongMemEval evaluation code.