A Comparative Study of Performance Measures for Information Retrieval Systems

Traditional performance measures of information retrieval systems include precision and recall and their variants. While these measures work well in closed-laboratory environments, they are not suitable for practical IR systems such as Web search systems. Many single-value measures were proposed to improve over the precision-recall measure, such as expected search length (ESL), average search length (ASL) and RankPower. We compare in this paper the measures of ESL, ASL, and RankPower applied to a set of real Web retrieval data. The results demonstrate that RankPower indeed is a feasible, effective, and easyto- use single-value measure for performance of practical IR systems such as Web search engines.