Investigating the research output of institutions

• There are manifold research results in informetrics, scientometrics, and research evaluation on the output of research institutions. What is missing is research on the completeness and reliability of the empirical basis of such investigations, which is the topic of this study. • One can search for the names of the institution under investigation or for the names of its research staff in a given time interval. • In order to make the quality differences between institution and author search more visible, we have introduced two variants of recall and precision considering on the one hand only the document base of the information service used and on the other hand a gold standard including all known publications of the institution. • There is a clear result: A search for affiliated authors outperforms a search for institutions in Web of Science. • Both search strategies ask for profound knowledge on the institution and its staff and a very high level of effort in data acquisition and processing. Describing, analyzing, and evaluating research institutions are among the main tasks of scientometrics and research evaluation. But how can we optimally search for an institution's research output? Possible search arguments include institution names, affiliations, addresses, and affiliated authors’ names. Prerequisites of these search tasks are complete lists (or at least good approximations) of the institutions’ publications, and—in later steps—their citations, and topics. When searching for the publications of research institutions in an information service, there are two options, namely (1) searching directly for the name of the institution and (2) searching for all authors affiliated with the institution in a defined time interval. Which strategy is more effective? More specifically, do informetric indicators such as recall and precision, search recall and search precision, and relative visibility change depending on the search strategy? What are the reasons for differences? To illustrate our approach, we conducted an illustrative study on two information science institutions and identified all staff members. The search was performed using the Web of Science Core Collection (WoS CC). As a performance indicator, applying fractional counting and considering co-affiliations of authors, we used the institution's relative visibility in an information service. We also calculated two variants of recall and precision at the institution level, namely search recall and search precision as informetric measures of performance differences between different search strategies (here: author search versus institution search) on the same information service (here: WoS CC) and recall and precision in relation to the complete set of an institution's publications. For all our calculations, there is a clear result: Searches for affiliated authors outperform searches for institutions in WoS. However, especially for large institutions it is difficult to determine all the staff members in the time interval of research. Additionally, information services (including WoS) are incomplete and there are variants for the names of institutions in the services. Therefore, searching for institutions and the publication-based quantitative evaluation of institutions are very critical issues.