Bibliometric indicators: opportunities and limits

Evaluating scientific research has always been difficult. The peer-review process, which has been the mainstay of science evaluation for nearly a century, takes time, expertise, and no small amount of resources to do properly. But several trends in scientific research have made this process even more challenging. The sheer number of scientific publications produced per year has been growing at an exponential rate for over fifty years 1, 2 and has shown no sign of slowing down anytime soon. These publications are also growing increasingly technical and specialized, making qualified reviewers more and more difficult to find. Finally, the glut of researchers in the biomedical pipeline combined with the recent recession have resulted in a larger number of researchers competing for a shrinking pool of available research funds 3. Evaluating scientific research in this context is becoming not only increasingly difficult, but also increasingly important to ensure that the right researchers receive promotions and funding to continue their work. In this environment, a number of review boards, institutions, and even countries are turning to bibliometrics to facilitate the review process. Bibliometrics is the quantitative analysis of publications. It essentially extracts data from publications and analyzes that data in various ways to answer questions about the research that those publications represent. It is a method of studying the producers, processes, and evolution of research using research publications as a proxy for research. As such, the field of bibliometrics encompasses a wide variety of approaches and methods, but it has become best known for its attempts to measure the impact of scientific research through the use of various bibliometric indicators like the impact factor 4 and the H-index 5. Because these indicators are perceived to be more objective than peer review, because they can be calculated with far less time and effort than peer review, and because there is some evidence that these indicators agree with peer judgment, reviewers and policy makers are increasingly using these indicators in addition to, and in some cases in place of, peer review of research impact. Although the use of bibliometric indicators can provide a valuable supplement to the peer-review process, these indicators are all too often taken out of context and applied without a full understanding of the bibliometric research on which they are based. As a result, they are frequently used to measure things that they were not intended to measure or to make comparisons they are not actually capable of making. This article provides a short introduction to the basic ideas behind these indicators and discusses ways that they can be used responsibly to minimize the biases of peer review. For more extensive and technical overviews of this topic, see Haustein 6 and Mingers 7.