Opening the Black Box of ‘Relevance’

The purpose of this project was to identify variables thought to affect relevance judgments and conduct a series of laboratory studies to determine the effects of these variables on relevance judgments. This paper discusses the variable of ‘implicit use orientations’—the particular attitude taken by a subject (judge) about the intended use of a document. One hundred and forty judges rated each of nine abstracts for relevance to several short information requirement statements. The same judges then repeated the ratings, each adopting (assuming) one of fourteen use orientations described to them. It was found that the particular use orientation assumed by the judge has a marked effect on relevance judgments. It was also found that implicit use orientations can be analysed in terms of their underlying structure and that their study offers the possibility of discovering the conditions under which one person can accurately simulate and use the implicit use orientations of another. Relevance judgments have been used as a basis for measures designed to evaluate the effectiveness of information retrieval systems. These judgments have usually been accepted at face value and have not been subjected to critical scrutiny. There is reason to believe, however, that as ordinarily obtained, they may be unreliable and sensitive to a number of conditions of measurement that have not been carefully controlled in previous evaluation studies.