Answers are grounded in this paper and cite their sources.
Comments are made on several specific questionable aspects of experimental design in the "Second Cranfield project." Statistical reasoning is used to show that very likely there must have been several times as many relevant "document-question matches" as were actually found by the Cranfield searchers in the process of determining, before any retrieval tests, "all possible" relevance matches. The argument derives from the Cranfield data itself, and rests essentially on the curiously small overlap between two apparently independent methods of seeking relevant documents (namely, searching by students and bibliographic coupling).
No takes yet. Share an insight, caveat, or question.
Don R. Swanson (1971) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: