Information Retrieval Test Collections

Many retrieval experiments have been based on inadequate test collections, and current research is hampered by the lack of proper collections. This short review does not attempt a fully documented survey of all the collections used in the past decade: hopefully representative examples have been studied to throw light on the requirements test collections should meet, to show how past collections have been defective, and to suggest guidelines for a future ‘ideal’ test collection. The specifications for this collection can be taken as an indirect comment on our present state of knowledge of major retrieval system variables, and experience in conducting experiments.