Large Language Model (LLM) question answering (QA) is essential for various real-world applications. As the demand for LLMs in domain-specific, up-to-date, and private scenarios increases, equipping them with large external corpus for QA, referred to long-context LLM QA, becomes critical. However, existing methods struggle to balance the comprehensive retrieval of scattered information required for accurate QA with the high token costs of long-context processing. We identify two major causes of this bottleneck: (1) limited retrieval coverage under complex queries and large input scales, and (2) repeated document access enforced by rigid query-by-query processing. In this paper, we introduce a cost-effective Retrieval-Augmented Generation Sketch (RAG-Sketch) for long-context LLM QA. To enable more thorough information retrieval without exhaustive context traversal, we propose a new skyline retrieval module incorporating query decomposition into a pre-defined schema. This approach retrieves Pareto optimal chunks by jointly considering semantic similarity and the presence of key information within the decomposed queries. Based on retrieved chunks, we formally define a set cover chunk merging optimization problem and proposes a greedy algorithm with theoretical guarantees that effectively reduces redundant access. Our experiments on widely used long-context QA benchmarks demonstrate that RAG-Sketch improves prediction accuracy by 24.5% while reducing token costs by 58.5% compared with the state of the art.
Zhu et al. (Mon,) studied this question.