On selecting a measure of retrieval effectiveness

Abstract It is argued that a user's subjective evaluation of the personal utility of a retrieval system's output to him, if it could be properly quantified, would be a near‐ideal measure of retrieval effectiveness. A hypothetical methodology is presented for measuring this utility by means of an elicitation procedure. Because the hypothetical methodology is impractical, compromise methods are outlined and their underlying simplifying assumptions are discussed. The more plausible the simplifying assumptions on which a performance measure is based, the better the measure. This, along with evidence gleaned from ‘validation experiments’ of a certain kind, is suggsted as a criterion for selecting or deriving the best measure of effectiveness to use under given test conditions.