First 20 precision among World Wide Web search services (search engines)

Five search engines, Alta Vista, Excite, HotBot, Infoseek, and Lycos, are compared for precision on the first 20 results returned for 15 queries, adding weight for ranking effectiveness. All searching was done from January 31 to March 12, 1997. In the study, steps are taken to ensure that bias has not unduly influenced the evaluation. Friedmann's randomized block design is used to perform multiple comparisons for significance. Analysis shows that Alta Vista, Excite and Infoseek are the top three services, with their relative rank changing depending on how one operationally defines the concept of relevance. Correspondence analysis shows that Lycos performed better on short, unstructured queries, whereas HotBot performed better on structured queries.