Techniques for reliable and efficient evaluation of information retrieval systems on large-scale corpora

Evaluating retrieval systems following the Cranfield paradigm requires a document collection, a set of information needs, and relevance judgments that might be pooled in a shared task. Ideally, such evaluations are reliable in the sense that the observed results are stable in similar scenarios. However, for large document collections, it can be difficult to verify the reliability of an evaluation. Potential issues include (1) redundancy in the collection, (2) incomplete relevance judgments, and (3) high costs to (re-)run retrieval systems. In this thesis, we analyze four post-processing techniques to improve the efficiency of evaluations that reuse evaluation resources from large collections while maintaining reliability. Efficiency is improved by reducing the size of the document collection, and reliability tests help to verify the size-reduced evaluation setup. We experimentally validate our four post-processing techniques in the order in which they would normally occur in a shared task. First, detecting near-duplicates helps to address the issue of redundancy in a collection. We find that near-duplicates can substantially impact the reliability (effectiveness is overestimated) and the efficiency (around 30% of documents can be skipped) of evaluations. Second, we support the archiving of the retrieval systems that were pooled. Our experiments show that the archived retrieval systems can be re-executed for reliability tests. Third, we use corpus subsampling to remove documents from the collection that are unlikely to be retrieved for the evaluated information needs. We find that our best subsampling strategy reduces the size of the collection by a factor between 10 and 1000 while maintaining the reliability of evaluations. Fourth, we bootstrap effectiveness scores to account for incomplete relevance judgments and for the respective uncertainties in the evaluations.