Answers are grounded in this paper and cite their sources.
A methods paper on evaluating retrieval for an agent memory that changes while it is measured, in two parts kept apart. Retrospective. Five ways the ruler misled us while building a Portuguese-language retrieval benchmark: gold keyed by chunk id decayed under re-ingest; results the oracle never judged were scored as misses; an LLM judge claimed verbatim quotes it had paraphrased, and a citation check built on them lowered agreement with a second judge; an ideal built from the ranking measured ordering instead of retrieval; and a second judge rejected part of the gold as not answering its own question, and a repair search mostly found the accepted passage in another file. These are findings of development, made and repaired on the same queries. Prospective. A pre-registered test, frozen by the tag paper-a-prereg-v1.0 before it ran, on 120 new questions written by a model from passages drawn at random and kept when they passed fixed acceptance rules. Pooled minus strict nDCG@10 stayed above the registered margin. A blind human sample did not validate the LLM judge, which limits what the pooled figure can claim. The paper reports the pair of scores as sensitivity to the annotation protocol, not as an interval around a true value. The repository history was rewritten on 2026-11-08; the rewritten commits keep their dates but are not independent evidence of when they were made, and the originals are kept in a private mirror. Persistence of references. The second registered hypothesis, that a reference by evidence keeps reaching its passage when a reference by chunk id does not, could not be tested: before the study began, the system had stopped replacing every id of a file on each re-ingest. One more reading is registered for 2027-05-18 (tag paper-a-h-a2-amendment-v1.0). A retrospective re-registration over six earlier copies of the database (tag paper-a-h-a2r-prereg-v1.0), designed with knowledge of that window, holds on its primary reading, conditional on the evidence surviving, and not over all references drawn. The corpus is private and is not released. The code-and-data archive carries the pre-registrations and the amendment, every script, both commitments, the printed outputs of each reading, and a provenance file that names the source of every number in the text.
No takes yet. Share an insight, caveat, or question.
Luiz Antonio Busnello (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: