Other people's judgments: A comparison of users' and others' judgments of document relevance, topicality, and utility

The emerging user-centric model of relevance proposes that the only valid measure of relevance of a document to a user's information need is the one made by that user. If we accept this proposition, it raises an interesting question: how well do other people, especially those involved in information work who make such judgments as part of their training and work, perform as judges of documents for information needs they did not originate? This question was empirically tested, using three groups of subjects: incoming students to a school of information/library science, continuing students in that school, and academic librarians (holders of the MLS degree). These subjects made judgments of either “relevance,” “utility,” or “topicality” of two document sets to the original users' stated information need. These judgments were then compared to those of the users to see what patterns emerged, and to see what can be learned not only about secondary judgments in general, but also the ways in which information and library professionals make such judgments. These results are interesting in their own right (subject's judgments compared reasonably well to those of users, looked more like users' after more training and experience in library work, and fall into interesting patterns), but they also lead to some provocative questions about the nature of judgment and evaluation of information items. © 1994 John Wiley & Sons, Inc.