Information retrieval evaluation based on the pooling method is inherently biased against systems that did not contribute to the pool of judged documents. This may distort the results obtained about the relative quality of the systems evaluated and thus lead to incorrect conclusions about the performance of a particular ranking technique.
No takes yet. Share an insight, caveat, or question.
Büttcher et al. (2007) studied this question.
Synapse has enriched 4 closely related papers on similar clinical questions. Consider them for comparative context: