Ranking, relevance judgment, and precision of information retrieval on children's queries: Evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

This study employed benchmarking and intellectual relevance judgment in evaluating G oogle, Y ahoo!, B ing, Y ahoo! K ids, and Ask K ids on 30 queries that children formulated to find information for specific tasks. Retrieved hits on given queries were benchmarked to G oogle's and Y ahoo! K ids' top‐five ranked hits retrieved. Relevancy of hits was judged on a graded scale; precision was calculated using the precision‐at‐ten metric ( P @10). Y ahoo! and B ing produced a similar percentage in hit overlap with G oogle (nearly 30%), but differed in the ranking of hits. Ask K ids retrieved 11% in hit overlap with G oogle versus 3% by Y ahoo! K ids. The engines retrieved 26 hits across query clusters that overlapped with Y ahoo! K ids' top‐five ranked hits. Precision ( P ) that the engines produced across the queries was P = 0.48 for relevant hits, and P = 0.28 for partially relevant hits. Precision by Ask K ids was P = 0.44 for relevant hits versus P = 0.21 by Y ahoo! K ids. B ing produced the highest total precision ( TP ) of relevant hits ( TP = 0.86) across the queries, and Y ahoo! K ids yielded the lowest ( TP = 0.47). Average precision ( AP ) of relevant hits was AP = 0.56 by leading engines versus AP = 0.29 by small engines. In contrast, average precision of partially relevant hits was AP = 0.83 by small engines versus AP = 0.33 by leading engines. Average precision of relevant hits across the engines was highest on two‐word queries and lowest on one‐word queries. Google performed best on natural language queries; B ing did the same ( P = 0.69) on two‐word queries. The findings have implications for search engine ranking algorithms, relevance theory, search engine design, research design, and information literacy.