A Blueprint of IR Evaluation Integrating Task and User Characteristics

Traditional search result evaluation metrics in information retrieval, such as MAP and NDCG, naively focus on topical relevance between a document and search topic and assume this relationship as mono-dimensional and often binary. They neglect document content overlap and assume gains piling up as the searcher examines the ranked list at greater length. We propose a novel search result evaluation framework based on multidimensional, graded relevance assessments, explicit modelling of document overlaps and attributes affecting document usability beyond relevance. Document relevance to a search task is seen to consist of several content themes and document usability attributes. Documents may also overlap regarding their content themes. Attributes such as document readability, trustworthiness, or language represent the entire document’s usability in the search task context, for a given searcher and her motivating task. The proposed framework evaluates the quality of a ranked search result, taking into account the contribution of each successive document, with estimated overlap across themes, and usability based on its attributes.