How the overlap between the search results of different retrieval systems correlates with document relevance

This paper provides an analysis of the overlap between the search results of retrieval systems that participated in TREC 3, 6, 7 and 8 to provide empirical support for some of the key assumptions guiding the design of data fusion methods. It shows that the potential relevance of a document increases exponentially as the number of search methods retrieving it increases — called the Authority Effect. It also shows that documents higher up in ranked lists and found by more systems are more likely to be relevant — called the Ranking Effect. Both effects can help explain why major data fusion methods perform well.