A Test for the Separation of Relevant and Non‐relevant Documents in Experimental Retrieval Collections

Many retrieval experiments are intended to discover ways of improving performance, taking the results obtained with some particular technique as a baseline. The fact that substantial alterations to a system often have little or no effect on particular collections is puzzling. This may be due to the initially poor separation of relevant and non‐relevant documents. The paper presents a procedure for characterizing this separation for a collection, which can be used to show whether proposed modifications of the base system are likely to be useful.