Information Retrieval as a Trial-And-Error Process

Recognition of the essential role of trial and error in access to scientific literature may point the way toward improved information services and may illuminate inconsistencies that have beset many retrieval experiments. This paper examines three important and well-known information retrieval experiments, with a focus on certain internal inconsistencies and on the high variability of search results. In these experiments, retrieval systems are evaluated in terms of their ability to select relevant documents and reject those that are irrelevant. It is suggested that this criterion is inadequate because of ambiguities inherent in the concept of relevance and that closer attention to trial-and-error processes may be helpful in developing better criteria. Specific examples of how one might improve document retrieval, library use, and citation indexing are offered.