Answers are grounded in this paper and cite their sources.
Algorithmic evaluation demonstrates a 23% performance variation across information retrieval systems, indicating that accounting for user findability effort alters assessment accuracy.
A user expresses their information need in the form of a query on an information retrieval (IR) system that retrieves a set of articles related to the query. The performance of the retrieval system is measured based on the retrieved content to the query, judged by expert topic assessors who are trained to find this relevant information. However, real users do not always succeed in finding relevant information in the retrieved list due to the amount of time and effort needed. This paper aims 1) to utilize the findability features to determine the amount of effort needed to find information from relevant documents using the machine learning approach and 2) to demonstrate changes in IR systems' performance when the effort is included in the evaluation. This study uses a natural language processing technique and unsupervised clustering approach to group documents by the amount of effort needed. The results show that relevant documents can be clustered using the k-means clustering approach, and the retrieval system performance varies by 23%, on average.
No takes yet. Share an insight, caveat, or question.
Rajagopal et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: