This paper gives an overview of the Web Retrieval Task that was conducted from 2001 to 2002 at the Third NTCIR Workshop. In the Web Retrieval Task, we attempted to assess the retrieval effectiveness of Web search engine systems using a common data set, and built a re-usable test collection suitable for evaluating Web search engine systems. With these objectives, we constructed 100-gigabyte and 10-gigabyte document data that were mainly gathered from the `.jp' domain. Participants were allowed to access those data only within the `Open Laboratory' located at the National Institute of Informatics. Relevance judgments were performed on the retrieved documents, which were written in Japanese or English, by considering the relationshiop between the pages referenced by hyper-links. Some evaluation measures were also applied to individual system results submitted by the participants.
Eguchi et al. (Tue,) studied this question.