Peer Review: A System under Stress

The American Institute of Biological Sciences (AIBS) convened a meeting in Washington, DC, on 6 December 2016 to explore The Role of Peer Review in Informed Decision-making. The participants included representatives from AIBS member organizations, government agencies, research funders, scholarly publishers, and scholars of peer review. In recent years, there have been a number of high-profile examples of flawed or fraudulent research being published. Simultaneously, many funders have been pressed to limit the amount of funds expended on peer review. Adding to these challenges have been an increase in the number of for-profit journals, the frequency of requests of scientists to review manuscripts, and interdisciplinary research that can strain the capacity of peer-review panel members. Peer-review managers have also been challenged to reduce bias and maintain the integrity of the process. To further strain the system, policymakers and the public increasingly demand transparency and open access to data and publications. Despite some flaws, the peer-review system has served us well. Of significant concern, however, is a growing belief that the system is under stress. Agencies are struggling to review increasing numbers of research applications, and journals are struggling to find a third reviewer for articles. The scientific community must allocate more attention to these issues, particularly to the need for improved training, new expectations for scientific community members, and greater research into the factors that inform and influence peer review. The scientific community regularly discusses the importance of research and education for improving the human condition through the creation of new knowledge; informed decision­making; and the development of new applications that enhance human health, environmental management, and food security. It is with less frequency that we consider the elements of the scientific process that stimulate or hinder these outcomes. The peer-, or merit-, review process is considered by the overwhelming majority of the scientific community to be an important tool for identifying quality science. It is through our community policing and review practices that we identify important work, sleuth out the rare—but present—huckster, and fuel the engines that power the creation of new knowledge. Despite our reliance on peer review, the factors that influence the process and its outcomes are not well understood (Rennie 2016). When it comes to scientific peer review, relatively few scholarly reviews of the factors that influence decision-making have been published. Importantly, this lack of information has led some lawmakers to question the validity of peer review. On 6 December 2016, AIBS convened a meeting of leaders from scientific societies and research organizations, journal publishers and editors, funding organizations, and government for an open dialog intended to improve our collective understanding of the status of peer review. The workshop-style meeting considered the following four overarching questions: (1) Do we know what we need to know about peer review? (2) Does peer review meet the needs of the modern publication system? (3) How effectively does peer review identify the best research proposals? (4) Is peer review properly incentivized, and are reviewers being asked to evaluate the right things? To set the stage for the day's deliberations, Dr. Drummond Rennie provided a historical overview of peer review in the United States. In addition to drawing attention to situations where peer review of both funding applications and publications has worked less than optimally, he also highlighted efforts by funders, editors, and scholars to conduct rigorous investigations into peer review. The balance of the meeting featured panel and group discussions. Because the purpose of this meeting was to foster an open dialogue, this report does not attribute specific remarks to individual meeting participants or sponsoring organizations. Rather, the following section provides a general synthesis of the discussions. A general recognition shared by scientists and managers of peer-review processes alike is that the system is under stress. If the strain is not ameliorated, there is a real concern about how much longer it can function before its foundation is significantly damaged. AIBS President Dr. Joseph Travis offering opening remarks. Photograph: Julie Palakovich Carr. Notably, journal editors are increasingly challenged to secure an adequate number of reviewers for publications, particularly in smaller, more specialized fields. Some hypothesize that the proliferation of journals and demands for rapid publication are taxing the community's capacity to provide reviews. As scientists increasingly turn down invitations to review because of limited time, many journal editors are also beginning to worry that the reviewers who are conducting these evaluations may not be allocating the requisite time to provide a high-quality analysis of a paper. Supporting this concern are high-profile examples of papers being published with seemingly obvious errors or misrepresentations or inaccurate presentations of data (Bohannon 2013). Funders report that they are still able to recruit scientists to participate on review panels because they are either compensated or because individuals value the insights they obtain from participating in the review ­process. However, if current trends continue with respect to the growth in the number of proposals with no growth in funding, this system may soon experience some of the problems now vexing journal editors and publishers. We know of some persistent challenges facing the peer-review system, such as bias, but some argue that the scholarly community continues to struggle to access the data required to identify other issues in the system or to test potential remedies. Government regulations, privacy concerns, and other issues limit access to government data sets about peer review, which are often the most robust and therefore offer great potential for research. Although evaluating the impact of an applicant's work based on citations has merit, the double-edged blade of this system is that it creates a cycle of funding productive, well-funded investigations at the expense of investments in new investigators (Rockey 2014). Individuals from historically underrepresented institutions or programs with a less prestigious reputation for research may also struggle to access resources under this model (Ginther 2011). Some have suggested that the Howard Hughes Medical Institute model of funding individuals rather than ideas might be a mechanism to relieve some of the stress on the system (Ioannidis 2011). This model, however, has challenges, and much care must be taken to not institutionalize bias into the funding criteria. The anonymization of gender, affiliations, and institutions in research applications could help reduce biases and would allow reviewers to focus on real rather than perceived or inferred competency, but this practice is not widespread and could potentially increase the cost of managing a peer-review process. As with research proposals, bias is an issue in the review of publications. In brief, researchers have found biases in favor of authors from prestigious institutions (Lee 2013). Once again, anonymization may address this but thereby introduce a significant new burden and management challenges into a system that is already under stress. Scholars of peer review note that there is a need for more active exploration of the measures of critique quality and how reviewer demographics, review setting (e.g., in-person or virtual), research field, and so forth affect the quality of a review (Carpenter 2015, Gallo 2016). Additional research is required to better understand the influence of funding success rates. For example, the decision-making processes of reviewers may be less prone to bias if it is known that a higher proportion of the proposals on their panel are going to be funded, because reviewers may be more forgiving in their evaluations. More investigations are also needed to better understand the reliability of reviews. In short, how many reviewers are actually needed to get a reliable assessment (Snell 2014)? Peer review of scientific papers has existed in one form or another for centuries. As the world has changed, so, too, has peer review. Importantly, the way we find and consume information has changed at unprecedented rates in just the past half century. Therefore, it seems reasonable to expect that we may need to consider alternative peer-review models now and in the years to come. Drs. Betsy Myers, DDCF, and Richard Nakamura, NIH, exchanging ideas during a panel discussion. Photograph: Julie Palakovich Carr. As science becomes more complex and increasingly draws on large data sets and collaborative teams spanning disciplines, institutions, and geopolitical borders, we must ask whether three reviewers and an editor have the requisite skills to thoroughly vet a manuscript—particularly when, as noted above, time is a considerably limited resource. Therefore, it may be important to consider the following: What does the modern publication system want? According to information gathered by Oxford University Press, authors have consistently reported that the purpose of peer review is for improving the quality of the published paper, determining the originality of the manuscript, determining the importance of the findings, detecting fraud, and detecting plagiarism. However, high-profile examples of where the system has failed to meet these objectives have captured widespread attention (Bohannon 2013). According to a recent study published in PLOS ONE, it has been estimated that 63.4 million hours were devoted to peer review in 2015 (Kovanis 2016). Despite the fact that they calculated that 20% of the researchers performed 69%–94% of the reviews, the authors argue that the system is sustainable with respect to volume but that there is a considerable imbalance in the distribution of the peer-review effort across the scientific community. Although the system may be technically sustainable, scientists and editors describe it as strained, potentially more so than the peer review of research applications. Increased recognition for reviewing manuscripts, new models for rewarding reviews, and additional financial resources to support the system may be required. Funders are routinely asked to—and are genuinely interested in being able to—point to the outcomes of the researchers they have funded (Gallo 2014). Indeed, the point of funding research is to expand our knowledge and to solve societally important problems. In some cases, there are outcomes of a research endeavor that clearly demonstrate the return on the research investment. In most cases, particularly in fundamental research, the outcome is more nebulous. We may have learned that something does not work. Although less captivating than some discoveries, these findings are important. A problem with these findings, however, is that most suspect they are published with far less frequency (Dwan 2008). Thus, metrics of research-funding impact are skewed. Aside from using citation indices, other measures of research impact should be considered. Productivity might also be assessed in terms of career progression (Escobar-Alvarez 2013), how the idea or research is assimilated into the field, effective dissemination of data (e.g., for research that is not published), resultant patents, and so forth. Algorithms could be developed to measure the outcomes and life of a research idea. The question of how or whether to incentivize reviews for articles or proposals is challenging. Budgets are tight and not expected to grow in the near term. Therefore, financial incentives come at the cost of either funding or publishing more research. Moreover, some journal editors who have thought about or experimented with providing financial rewards to reviewers urged caution because the efforts were expensive and did not seem to provide an adequate incentive to encourage individuals to review manuscripts. An alternative incentive model suggested, particularly for research-application reviewers, was to explore a process in which a program officer identifies good reviewers (e.g., timely, responsive, and thorough) and enters these individuals into a lottery for research funds. Another model, which would be appropriate for publication and proposal review, is to once again ensure that this “community service” is properly acknowledged and rewarded through professional evaluation, promotion, or tenure procedures. One suggestion offered for ways to enhance the peer review of research funding proposals was to “personalize” the nature of the review forms provided to panelists to reflect their expertise. In other words, modify review forms so that a statistician is being asked to evaluate a proposal's statistical elements and not whether it adequately engages a citizen scientist. For programs that include external stakeholders (e.g., patient advocates) on review panels, adjust the forms to ensure that they are capturing this individual's expertise. There was agreement that peer review has served us well and no one has yet identified a viable alternative. However, following the day's discussions, deliberations by four breakout groups, and a summary group discussion, the participants offered a number of calls for action. There are an ever-increasing number of pressures and demands on the time of researchers. As a result, they lack the capacity to participate in the peer-review process in the same way or with the same investment as they did 20 years ago. It is therefore important that the efforts of individuals who engage in the peer-review process are appropriately acknowledged and incentivized. The idea of financially compensating individuals for reviewing manuscripts was suggested, but individuals who have experimented with this model urged caution, noting that it is expensive and does not necessarily produce the desired outcome. Other forms of recognition, such as from home institutions and professional communities, may be more successful. Given growing time constraints and changes to how we access, process, and use information, it would be wise for there to be an effort to cultivate a more vibrant scholarly community focused on understanding how different factors influence peer-review processes, whether related to funding applications or publications. Moreover, there is a need for additional scholarship that can inform the development of best practices in peer review. Efforts to reduce bias, identify the most impactful proposal, or best manage an interdisciplinary review panel, among other objectives, would be best informed by data. Scholars of peer review would welcome opportunities to partner with organizations with access to significant data sets, such as federal agencies, to stimulate new research. Confidentiality and other concerns could certainly be addressed via contractual agreements. Importantly, the lessons learned from these investigations would provide valuable insights to guide professional-development programs that would ensure that scientists, journal editors, and program officers have similar or shared perspectives as to what constitutes effective peer review. Peer-review professionals, the individuals charged with operating these systems, would also then be better equipped to make informed decisions about the application of different peer-review models or inventive structures. A stark realization was that most scientists, including editors and program officers, generally receive limited to no professional training about peer review. Most members of the scientific community essentially learn on the job. If peer review is as central to the quality of science as we report, the scientific community should invest more energy ensuring that all community stakeholders are appropriately trained and prepared to effectively participate in the peer review of scholarly communications and research-funding applications. The American Institute of Biological Sciences (AIBS) gratefully acknowledges the generous financial support for this meeting that was provided by the following organizations: the American Society of Parasitologists; Burk and Associates, Inc.; the Burroughs Welcome Fund; the Doris Duke Charitable Foundation; the Ecological Society of America; Oxford University Press; and the Society for Integrative and Comparative Biology. Although these organizations supported this meeting, the findings and recommendations do not necessarily reflect the positions or opinions of any of these groups or of the individuals who participated in this meeting. We also extend our appreciation to the AIBS staff members who were instrumental in planning and supporting this meeting. Specifically, we commend the dedicated efforts of Diane Bosnjak, Julie Palakovich Carr, Syreeta Jones, Jeff Newman, and Joel Wagner.