Order effects: A study of the possible influence of presentation order on user judgments of document relevance

Studies concerned with the evaluation of information systems have typically relied on judgments of relevance as the fundamental measure in determining system performance. In most cases, subjects are asked to assign a relevance score using some category rating scale (1–4, 1–11, or simply relevant/non-relevant) to each document in a set retrieved in response to some information need or query. While the extensive studies of relevance conducted in the 1960s indicated that relevance judgments are influenced by a range of variables, little attention has been paid to the possible effects of the order in which the stimuli are presented to judges. This effect of “stimulus order” has been found to exist in measuring variables in other fields (Stevens 1975, Gescheider 1985). Questioning possible “presentable order effects” is particularly appropriate in that systems are being developed and evaluated in information science which present documents in some systematic way (e.g., with the documents considered by the system to be most relevant presented first). This article describes an effort to study whether the order of document presentation to judges influences the relevance scores assigned to those documents. A query and set of documents with relevance judgments were available from a previous study. Subjects were randomly assigned one of two orders (one ranked high to low, the other low to high) of fifteen document descriptions. They were then asked to assign a score to each document description to match their judgment of relevance in relation to the stated information query. Both a category rating (1–7) and open-ended, magnitude estimation scaling procedure were tested, and it was found that the judgments were influenced by the order of document presentation. © 1988 John Wiley & Sons, Inc.