GenTREC : The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems

Building test collections for Information Retrieval evaluation has traditionally been a resource-intensive and time-consuming task due to reliance on manual relevance judgments. While various cost-effective strategies have been explored, the development of such collections remains a significant challenge. This paper introduces GenTREC , the first test collection built entirely from documents generated by a Large Language Model (LLM), eliminating the need for manual relevance judgments. Assuming LLM-generated documents are inherently relevant to their prompts, we used TREC search topics to generate documents, considering a document relevant only to its generating prompt. To introduce realistic retrieval challenges, we also generated non-relevant documents, ensuring that IR systems are tested against a diverse and robust set of materials. The resulting GenTREC collection comprises 96,196 documents and 300 topics. We conducted extensive experiments to evaluate GenTREC in terms of document quality, relevance judgment accuracy, and evaluation reliability. Our results show that IR system rankings using GenTREC align with traditional TREC test collections for P@100, MAP, RPrec, and nDCG metrics. Our findings demonstrate that GenTREC offers a promising, low-cost alternative for IR evaluation, significantly reducing the burden of building and maintaining future IR evaluation resources.