Karpathy’s proposal to replace Retrieval-Augmented Generation (RAG) with plain-text knowledge bases maintained directly by language agents (Agentic Knowledge Curation) has gained traction in industrial applications yet lacks systematic empirical evaluation. To our knowledge, this study presents the first comparative evaluation of this paradigm using blind human assessment by two independent external reviewers. The corpus comprises the technical documentation of a production semantic search system for Spanish legal documents (11 files, ~19,300 tokens) alongside 100 questions verified against source code, distributed across direct retrieval, multi-document synthesis, and reasoning about absence. Claude 3.5 Sonnet was utilized as the reference model to isolate the retrieval architecture’s effect. While overall accuracy was statistically indistinguishable between paradigms, agentic curation showed a significant advantage in direct retrieval over long documents. Conversely, error analysis revealed RAG false negatives associated with the lost-in-the-middle phenomenon, whereas agentic curation exhibited localized degradation in textual fidelity for queries requiring exact reproduction of formulas or sequences. These results characterize the differential error profiles of both paradigms, providing actionable design criteria for engineers managing technical documentation with language models in cloud environments.
Martín et al. (Mon,) studied this question.