Free for Service: The Inadequate Incentives for Quality Peer Review

“No man but a blockhead ever wrote except for money,” Samuel Johnson said. On the other hand, according to the Thomson Reuters’ Journal Citation Reports®, journals published a total of 11,291 orthopaedic articles in 2010, and most of the authors, we can safely assume, were neither blockheads nor paid for the piece. Was Johnson wrong? Not necessarily. One could make the argument that authors were paid indirectly. Publication can lead to grant support or prompt a bonus from an academic department. Beyond that, there are nonpecuniary rewards, such as recognition in the media or, more valuably, earning the esteem of one’s colleagues. Even with those examples in mind, Johnson’s larger point — that writing requires incentives — still holds true. Therefore, it is reasonable to ask whether the incentives for academic writing are calibrated correctly. If the rewards are too meager, we face a deficit. If the rewards are too great, excesses abound. There are data to suggest a surplus of academic writing. Of those 11,291 orthopaedic articles written in 2010, more than one-third were not cited as of January 2013, according to the Thomson Reuters’ Journal Citation Reports®, there are other metrics of merit besides the number of times a researcher cites a paper, but that so many papers came up empty suggests a problem. The low rates of citation may reflect shortcomings of the peer review process: either too many lower quality papers pass the filter of peer review, or reviewers failed to wring out the high-quality essence from the manuscripts under consideration. And if the review process is coming up short, it is reasonable to circle back to Johnson’s point and ask whether the incentives for high-quality peer-review are sufficient. Perhaps no man but a blockhead would ever write a high quality peer review except for money (or other similarly powerful rewards). Of course reviewers are not blockhead; still I would argue that the incentives for quality peer review are inadequate to the task. Peer review is hard work. The reviewer must grapple with the soundness of the methods, the validity of the results, and the reasonableness of the conclusions. References must be checked and cited papers may need consultation. Thereafter, the reviewer must write a dispassionate critique, with empathy for the writer and future readers alike. This can be terribly taxing. In return for this work, the reviewer gets … nothing. Well, not exactly nothing. Reviewers can expect the gratitude of the editor; the satisfaction of doing their civic duty as scholars; the power of shaping the medical literature (thereby helping patients); and (one hopes) the inherent pleasure from a job well done. Still, that may not be enough. One way to create incentives for greater participation and better reviews would be to allow reviewers to sign their work, and to publish the review alongside the paper itself. This not only creates a system of accountability, it allows the reviewers to earn a publication credit as well. If an entry on the curriculum vitae motivates the authors, a similar incentive should motivate the reviewers, too. Additionally, these pieces of commentary are likely to be appreciated by the readers. In the end, though, a system of signed reviews is not apt to take root, for at least two reasons. First, taking away the veil of anonymity may take with it the brutal frankness the job requires. And if that were not enough (and I think it is), reviewers will not be “paid” with a publication unless and until the paper under review is published as well and that creates a perverse incentive for a reviewer to support the acceptance of the marginal manuscript. Another way to create incentives for greater participation and better reviews is to grant peer-reviewers continuing medical education (CME) credit for their work. CME credit is perfectly appropriate to the task. The close reading of a manuscript is much more of an educational endeavor than passive listening in a lecture hall, to say nothing of web-surfing in a lecture hall or kibitzing in the hallway outside the lecture hall — and CME credit has been earned for all of those. (Don’t ask me how I know). While providing CME credit may impose costs on the journal, it provides substantial value to the recipient, as most physicians are required to collect CME credits to maintain their state licenses. Along those lines, perhaps the American Board of Orthopaedic Surgery would be willing to deem a reviewer’s record as “Evidence of Professional Standing.” A new approach, in which CME credit is granted to peer reviewers, qualifies as what economists term a “Pareto improvement”: a change that helps some and harms no one. With the advent of new publishing models such as open access journals and Internet-based publications, we can expect a wave of more academic papers in the years to come. As such, the need for high-quality peer-reviewed papers will be even more acute. To help meet that rising demand, better incentives for peer-review may be needed. Providing CME credit for this work is a worthwhile first step. Commentary Elizabeth Wager PhD Publications Consultant and chair of the Committee on Publication Ethics (COPE) Bernstein argues that orthopaedic literature could be improved if peer reviewers were better rewarded for their labors. In particular, he suggests peer review should be recognized by CME credits. This is an interesting idea, but not a new one. Several journals, including JAMA, the NEJM and Annals of Internal Medicine have been doing it since 2004 3, the Archives of Neurology since 2009 12, and the European Respiratory Journal since 2010 11. Most offer credits only to reviews that meet certain quality criteria and are received on time. It is a shame that the effects of such innovations have not (as far as I can tell) been formally tested. While it might be difficult to perform a randomized trial, it should be possible to do a simple “before-and-after” study to measure, for instance, whether offering credits for timely reviews actually improves punctuality. Presumably, the fact that some major journals have been offering CME credits for almost 10 years, suggests they find it beneficial, but it would be nice to see the evidence. Researchers from the University of Brescia, Italy 4, have adapted an online investment game to test the effects of incentive provision on the quality of peer review. Interestingly, the researchers conclude that editors and funders “should be extremely careful in offering material incentives” to reviewers “since these might undermine the moral motives which guide referees’ behavior” 4. Alternative models for recognizing reviewers have been proposed by researchers in India. Kumar 9 suggested that reviewers be offered coauthorship of articles, while Kachewar and Sankaye 8 proposed a “Reviewer Index” which, they hope, may raise the quality of peer review and recognize the reviewers’ contributions. While these variants on the peer review process are interesting, and should be tested, I propose a more radical solution to the problem underlying Bernstein’s suggestion. He notes that more than one-third of orthopaedic articles published in 2010 were not cited in the following 2 years. He suggests that, perhaps, the reviewers “failed to wring out the high-quality essence from the manuscripts.” I am afraid I take a more cynical view. I am not convinced that most manuscripts submitted to journals actually contain “high quality essence.” While I concede that peer review could be improved, even the best review cannot redeem a poorly designed study. Reviewing should be able to rectify poor reporting (although often it does not), but it cannot save under-powered or muddled research. Statisticians are fond of reminding us that, if they are consulted too late, rather than being involved in the study design, they are unable to cure a bad study, only to tell us “what it died from.” Similarly, peer reviewers cannot bring a dead study back to life. Offering reviewers CME credits might, have some benefits, and such initiatives should be tested. But to improve the quality of submissions, I believe we need to focus on better training in research methods and initiatives similar to the Standard Protocol Items: Recommendations for Interventional Trials 2 to raise the standard of protocols. James D. Heckman MD Editor Emeritus The Journal of Bone and Joint Surgery It is an honor to comment upon Dr. Bernstein’s ideas entitled “Free for Service: The Inadequate Incentives for Quality Peer Review.” While he clearly describes the intense effort that must be made to either write a scientific manuscript or to review it effectively, I believe he misses the most important reason for engaging in the process. The ultimate purpose of contributing to the scientific medical literature, whether that be by writing or reviewing, is to enhance the quality of life of our patients. During the past 35 years, I do not believe that I have written an article or edited a manuscript for any other reason than to advance the orthopaedic knowledge base. For me, sufficient incentive and reward has always been the knowledge that the thoughtful and conscience writing, reviewing, and publication of new ideas will be good for our patients. Financial remuneration should not be necessary or even considered as a reward. I applaud Dr. Bernstein’s recommendation to grant CME credit for reviewing manuscripts because doing so documents, in some small way, the substantial effort required to produce an effective review. It also acknowledges the individual’s participation in one of the most important phases of health care: the advancement of the field. In my opinion, the American Board of Orthopaedic Surgery should recognize such efforts as clear “Evidence of Professional Standing” because being selected as a reviewer represents the reviewer’s identification as an expert in the field. These are appropriate types of “compensation” for a reviewer’s efforts. Finally, I believe that a blinded peer review process is better than one with full disclosure of the reviewer’s identity to authors or to the entire readership. As with financial compensation for reviewing, I do not believe that reviewers should participate in this process in order to add citations to their curriculum vitae, and I do not believe that most promotions committees would value such citations very highly anyway. Blinded peer review allows for a more open and thorough critique of a submitted manuscript. The editor who has assigned the manuscript for review should be fully aware of the reviewer’s conflicts of interest and use that information in the final appraisal of the submitted review. In this way, bias and prejudice are minimized while all sides of the subject are frankly explored during the review process. John L. Zeller MD, PhD Department of Orthopaedic Surgery University of Michigan School of Medicine “Medical Essays and Observations,” published by the Royal Society of Edinburgh in 1731, is considered the first peer reviewed publication 1. The reasons for solicited peer review were prompted then, as they are now, in response to limited expertise and conflicting opinions. Despite the contemporary angst associated with this process 5, 6, I would contend that the current system has enduring value, ensuring that research passes certain standards of scientific quality and integrity. At its best, peer review adapts to new inputs, excludes misinformation, and raises the quality of the end product (critical for the advancement of medical education, and a requirement for establishing a reproducible body of global research). Despite the fact that peer review has such a long and well-established history, research delineating its effects is a recent phenomenon. Overt guidelines and standardized methodologies remain fragmentary 7. In response to these deficiencies, the Seventh International Congress on Peer Review and Biomedical Publication in Chicago 10 focused on a plethora of topics: editorial and peer review decision making, research and publication ethics, authorship and responsibility for published materials, publication misconduct, evaluation of the quality of print and online information, and models for peer review. Based upon the editorial by Dr. Bernstein, this response is deliberately fabricated on a pragmatic basis, an introspection hoping to make the current system (as it is) more responsive and agreeable to those who now take on the task as peer reviewers. Accordingly, I submit four recommendations that can be universally applied to medical peer review: Raise the threshold of the “first pass” based upon journal priorities. Assigning editors should consider a 50% to 75% rejection rate for submitted manuscripts. This will reduce the workload that will eventually be assigned to peer reviewers. Set guidelines in a concerted effort to rate its peer reviewers. With the acknowledgement that this is an arbitrary exercise, reviewers who are described as “fair and/or “poor” should be eliminated. This redefined designation will accentuate the academic legitimacy of those reviewers who are routinely asked to participate in the evaluation of manuscripts. Identify reviewers who played a significant role in the evaluation of the journals manuscripts every year. Letters of appreciation should go out to each reviewer serving as verification of the individual’s efforts related to their field of expertise. Offer three units of CME credit for editorial commentary; whereas, each manuscript reviewed should be commensurate with 10 to 12 units of CME credit based upon the individual’s time and effort. Such compensation for effort would be a welcome incentive for reviewers and will help satisfy the yearly requirement for CME for faculty status, subspecialty certification, and state licensure. Based upon the directives of the Seventh International Congress, experiments aimed at peer review will undoubtedly be undertaken. I look forward to the potential benefits that will be gleaned from such innovations and to the meta-analysis that must be subsequently conducted.