Relevance Judgments of Actual Users and Secondary Judges: A Comparative Study

This exploratory study sought to examine judgments of relevance of document representations to query statements made by people other than the originators of the queries. A small group of graduate students in the School of Information and Library Studies and undergraduates at the University of Michigan judged sets of documents that had been retrieved for and judged by real users for a previous study. The secondary judges' assessments of relevance were analyzed by themselves and in comparison to the users' assessments. The judges performed reasonably well, but some important differences were identified. Secondary judges use the various fields of document records differently than users, and they have a higher threshold of relevance. There are other interesting differences and similarities between the groups. Implications of these findings for designing and testing document retrieval systems are discussed.