How scientometric indicators shape relevance: an exploratory experimental user study

Citations and altmetrics are prominently displayed in academic search systems, yet their role in user-driven relevance evaluation remains insufficiently understood. This exploratory user study investigates whether and how scientometric indicators influence user attention and relevance judgments during academic literature search. In a within-subject experiment, 50 researchers from the life and social sciences selected relevant articles from predesigned search engine results pages (SERPs) displaying either citation counts, altmetric scores or no indicators. Indicator values were experimentally manipulated to examine their effects on user interaction and document selection. The qualitative analysis combines think-aloud protocols with quantitative interaction data (mentions, clicks, visits, selections). Results indicate that citation counts were widely recognized, underscoring their embeddedness in academic evaluation practices. High-value indicators increased interaction at intermediate stages of SERP interaction. Altmetrics, in particular, elicited more consistent interaction effects than citations. The think-aloud data suggest that this pattern may be linked to participants’ unfamiliarity with altmetrics, which led to less critical scrutiny. At the same time, the altmetric score was often treated as interchangeable with citations. Most mentions of both indicators lacked contextual interpretation and reflected their use as mere numerical cues in a generalized “more-is-better” heuristic. However, increased attention did not translate into lasting effects in final article selections. The findings suggest that scientometric indicators function less as decisive relevance clues than as visual triggers for attention. Search interfaces that prominently display scientometric indicators may shape information behavior not by informing relevance judgments, but by steering user attention in ways they do not consciously reflect on.