Evaluation of evaluation in information retrieval

Evaluation is a major force in research, development and applications related to information retrieval (IR). This paper is a critical and historical analysis of evaluations of IR systems and processes. Strengths and shortcomings of evaluation efforts and approaches are discussed. together with major challenges and questions. A limited comparison is made with evaluation in experts systems and Online Public Access Catalogs (OPACS). Evaluation is further analyzed in relation to the broad context and specific problems addressed. Levels of evaluation are identified and contrasted. most IR evaluations were concerned with the processing level, but others were conducted at the output, users and use, and social levels. A major problem is the isolation of evaluations at a given level. Issues related to systems under evaluation, and evaluation criteria, measures, measuring instruments, and methodologies are examined. A general point is also considered: IR is mcreasmgly imbedded into many other applications, such as the Internet or digital libraries. Little evaluation in the traditional IR sense is undertaken in relation to these applications. The challenges are to inte~ate IR evaluations from different levels and to incorporate evaluation in new applications. explosion is a complex social, cognitive, cultural, and communication problem, and not just a technical problem. Thus, at its base IR is an activity oriented toward problems with the same (social, cognitive, communication) characteristics. Ultimately, IR exists as a social activity to link 101-108.