Improving peer review: What reviewers can do

As an author, there is no better feeling than to receive correspondence from a journal editor that your paper has been accepted for publication. However, that sense of elation can quickly turn to dread if the author has to respond to dozens, even hundreds, of comments from peer reviewers. We have observed that the quality of peer reviews has declined over the past decade, and their focus is often misguided, emphasizing grammar and style over science and methodology. This commentary provides guidance to reviewers on the best practices for conducting a peer review for biomedical journals. Benefits of being a peer reviewer. Volunteering to be a peer reviewer is an important first step in the peer-review process. Reviewers are typically identified based on their expertise in a particular field. Peer reviewers provide benefit to authors, editors, and journals by providing useful guidance related to a scholarly work, evaluating the methodology and importance of the science, and making recommendations regarding publication decisions.1,2 There are benefits to serving as peer reviewers as well. These include keeping abreast of the latest developments in their field and improving their skills to critically evaluate scientific papers, which may improve their own scientific writing. Lastly, it is an honor and a privilege to be a member of a profession. Thus, there is a moral obligation to give back to one’s professional and scientific community.3,4 For these reasons, participation in peer review is a professional expectation in many settings. Characteristics of a good peer reviewer. Most biomedical journals use the feedback provided during the peer-review process to make editorial decisions on manuscripts submitted for publication. Typically, editors of peer-reviewed journals seek content experts to perform a critical analysis of the manuscript and provide (1) specific comments regarding the manuscript’s scientific merit, (2) recommendations to authors on how to improve the manuscript, and (3) recommendations to the editor regarding manuscript disposition. Invitations to serve as a peer reviewer are often accompanied by Web links to reviewer resources (e.g., reviewer guidelines), but formal training is usually not provided nor required. The characteristics of a good peer reviewer are not well defined. While several studies have attempted to identify characteristics that predict reviewer performance, there is little consistency in the findings, and the associations are weak.5,–9 Younger age of the reviewer is 1 of the only characteristics that has been consistently associated with stronger peer reviews.5,–8 Some researchers have found a positive association between the duration of time spent reviewing the article (up to 3 hours) and review quality.5,7 The quality of peer review varies greatly, perhaps partly because few professionals receive training in this area.1,10 However, formal training alone likely only marginally affects the quality of peer reviews. Investigators who conducted a study of more than 300 reviewers who performed 2,856 reviews for the Annals of Emergency Medicine did not find an association between more than 10 hours of training in critical appraisal skills and review quality.6 However, a BMJ study of 609 reviewers found a small, short-term benefit (<6 months) associated with participation in a peer-reviewer training program.9 Nevertheless, the variation in the quality of peer reviews suggests that peer reviewers may benefit from some guidance. What reviewers can do: A step-by-step approach. Although other approaches are acceptable, we recommend a step-by-step process for peer reviewers (Figure 1). Regardless of the approach used, the best peer reviews are systematic and comprehensive. Step-by-step process for performing a peer review.1,–4,10,–13 Responding to the invitation to perform a peer review. Peer reviewers should consider several issues before deciding to accept or decline an invitation to perform a peer review. Reviewers should first assess whether they are qualified to review the manuscript.2,10,11 Reviewers are typically provided an abstract of the manuscript with the invitation to review. Reviewers should only accept invitations to review if they have expertise in the manuscript’s area of focus. In some cases, the reviewer may have the expertise to review a key section of the manuscript (e.g., research methodology, statistical analysis) but not other sections. In this instance, the reviewer should be clear about the scope of the review before accepting the invitation.2,10 If the manuscript is within the reviewer’s area of expertise, potential or actual conflicts of interest should be considered.2,10,11 Potential reviewer conflicts may include close personal (e.g., spouse) or professional (e.g., mentor–mentee, institutional colleague) relationships with 1 or more of the authors.10,11 In addition, the presence of conflicting or competing interests that may create the potential for personal, financial, intellectual, professional, political, or religious benefit by the reviewer may be considered conflicts of interest.2,10,11 Reviewers should ask themselves this simple but powerful question: Would others—the authors, editors, or readers—believe that my positive or negative review of this paper is in any way motivated by self-interest? If the answer is yes, it is critical to discuss the potential conflict with the editor before accepting the invitation to review.2,10,11 Timeliness is critically important to the peer-review process. To avoid unnecessary delays in the publication process, the reviewer must consider his or her ability to meet the journal’s deadline for completing the review before accepting the invitation.2,10,11 A quality peer-review can take several hours to perform.3,5,7,10 Reviewers are typically given 2 weeks to complete their peer review, though the timeline may be as brief as 48 hours and as long as 4 weeks.11 Reviewers should respond to peer-review invitations in a timely manner, especially if the invitation is declined.2,12 In some cases, journals will rescind their invitations if a timely response (e.g., within 3–4 days) is not received. Throughout the process, reviewers must maintain confidentiality and should never circulate the manuscript to or discuss its contents with others.1,12 If reviewers would like to include others (e.g., trainees) in the peer-review process, permission should be requested to do so, or at a minimum, the editor should be notified when the critique is submitted.2,12 Performing the peer review. Quick read. Many agree that the first step in completing a peer review is to perform a quick read.1,3,4,10,13 During this first read, reviewers should skim the article to gain a basic understanding of the manuscript’s contents.13 During the quick read, reviewers should once again assess whether they have the expertise to perform the review and whether any real or potential conflicts of interest exist. Reviewers should notify the editor immediately if these problems are identified.4 Reviewers should note ambiguities and questions that arise during the quick read and begin to form opinions regarding the importance and relevance of the work.3,10 Another important reason for the quick read is to identify critical or “fatal” flaws of the manuscript.1,4,10,13 In general, a manuscript’s flaws can be categorized as minor (do not require author action), major (require satisfactory resolution before publication is considered), and fatal (deem the manuscript—even with major revisions—irreparable).1,10,13 If reviewers identify a fatal flaw during the quick read, they may recommend to reject the manuscript at this point, forgoing a more-detailed read and written critique.1,4,10,13 Common examples of manuscript flaws are listed in Table 1. Manuscript Flaws That May Warrant Revision or Rejection During Peer Review1,10,13 Manuscript Flaws That May Warrant Revision or Rejection During Peer Review1,10,13 Detailed read. After the quick read, reviewers should perform a more detailed, critical read of the manuscript.3,4,10,13 Some scholars have suggested that this occur hours to days after the first read, as this may allow the reviewer to better prepare for a more critical read by reviewing ancillary materials (e.g., reviewer instructions, figures, tables, supplementary files).2,3,13 The major purpose of the detailed read is to determine the scientific merits and relevance of the manuscript.3,4,13 To help formulate this evaluation, reviewers should ask themselves 3 questions about the manuscript: Do I understand it? Do I believe it? Do I care?11 Manuscripts should clearly state the research question or clinical problem and then describe how the scholarly work was conducted. If valid methods are used and the conclusions are supported by the data, the results should be credible. Finally, if the work is original, the question or problem important, or the intervention innovative, it should stimulate interest in the reviewer and readers alike. Reviewers should also note ambiguities and questions that arise during their detailed review and provide suggestions for improvement.3,4,13 Importantly, the reviewer should not focus on language, writing style, and grammar unless these affect the clarity of a particular point or the frequency of these issues significantly impairs the manuscript’s overall readability.3,4,11,13 In such cases, the reviewer should not attempt to rewrite the paper for the authors but rather point out the sections of the manuscript that need to be addressed. The detailed read should evaluate all aspects of the manuscript, including the title, keywords, abstract, introduction, methods, results, discussion, conclusion, graphs, tables, figures, and references. Most minor and major flaws (Table 1) are identified during this more-critical read. For research manuscripts, manuscript flaws are commonly found in the methods section. Reviewers should consider using a systematic process (e.g., Randomized, Attrition, Measurement by blinded assessors or objective measurements [RAMbo]) or follow reporting guidelines when critically evaluating randomized clinical trials (e.g., CONsolidated Standards of Reporting Trials [CONSORT]) or meta-analyses (e.g., QUality Of Reporting Of Meta-analyses [QUORUM]), particularly for reviewers with limited experience.1,4,14,–16 When evaluating the conclusion section, reviewers should ensure that the authors do not overstate their findings and identify any statements that may be perceived as biased. Although not an exhaustive list, Table 2 lists the characteristics of quality manuscripts worthy of acceptance. Reviewers should not be afraid to recommend conditional acceptance or rejection for those manuscripts that they believe have significant flaws or are insufficiently unique or important. Characteristics of Quality Manuscripts Worthy of Acceptance1,3,4,10,11,14 Characteristics of Quality Manuscripts Worthy of Acceptance1,3,4,10,11,14 Final scan. Reviewers should perform a final scan of the manuscript, focusing on organization, flow, and other issues that affect clarity.10 Reviewers should keep in mind that their assessment of the scientific merit of the manuscript is more valuable to the journal editors than their assessment of grammar and writing style.3,4,10,13 The final quick read also provides the reviewer an opportunity to highlight minor flaws (Table 1) and offer suggestions for improvement.1 Written critique. Reviewers should prepare their written critique to the journal editors by first summarizing the manuscript and then providing specific comments related to the identified flaws.4,10,11,13 Although some reviewers use the first paragraph of the written critique to summarize the manuscript’s contents and findings, we suggest that this summary focus on the reviewer’s overall analysis of the manuscript.4,10,11,13 When providing their overall analysis of the manuscript, reviewers should highlight the strengths, point out significant flaws, and comment on the importance or significance of the work.4,10,11,13 Throughout the critique, reviewers should be honest and direct but considerate when providing feedback to authors.1,13 Grammatical issues identified by the reviewer that are pervasive (e.g., numerous spelling errors) should be summarized, leaving detailed review of grammar, spelling, and writing style to the journal’s copyeditors and technical staff.4,10,11,13 The summary should balance both positive and constructive feedback consistent with the overall evaluation of the manuscript (e.g., the majority of comments should be positive if the recommendation is to accept or accept with minor revisions).4,10,11,13 If a reviewer reviewed only specific portions of the manuscript, this fact should be clearly stated in the opening summary.2 Following the summary, reviewers should provide specific comments related to each section of the manuscript. Reviewers should organize their comments in a logical manner and number them.10,11,13 In addition, reviewers should specify where within the manuscript (e.g., page and line numbers) the comments are directed.13 Many reviewers neglect to provide comments about the title and abstract, but these are perhaps the most important elements of the manuscript. Many readers will read or have access to only the title and abstract when conducting a literature search. It is helpful to the authors and the editors to indicate whether a comment identifies a major issue that must be addressed or is merely a recommendation (i.e., minor issue).4,13 Given that one of the purposes of peer review is to improve the quality of manuscripts, reviewer feedback should be objective and constructive, with actionable suggestions for addressing flaws.2,4,10,11,13 Because the classification of a manuscript’s flaws is subjective, if reviewers determine that a fatal flaw exists, an explanation of why the manuscript is beyond repair is warranted.1,10,13 When appropriate, feedback should be supported with evidence (e.g., supporting literature) to assist the editors in their decision and the authors in their revisions.2,4,10 When a statement in the manuscript is ambiguous and may lead to reader confusion, the reviewer may offer specific suggestions to improve clarity.4,10,11,13 Reviewers should refrain from divulging their publication recommendation to the authors in the critique. The responsibility to determine the manuscript’s disposition rests with the journal editors. Reviewers are often asked to provide confidential feedback to the editors. These comments should include a brief explanation to justify the reviewer’s publication recommendation.13 Other confidential comments that reviewers may wish to include are unvarnished opinions on specific aspects of the manuscript, concerns regarding ethics (e.g., plagiarism, fraud, unethical research practices), and the acknowledgment of individuals who may have assisted in conducting the review.1,4,11,13 Publication recommendation. In most instances, reviewers are asked to provide their recommendations regarding publication as follows: accept, accept with revisions, or reject. Some journals delineate “accept with minor revisions” from “accept with major revisions.” In general, manuscripts that are well written, novel, scientifically powerful, and relevant to the journal’s readers warrant acceptance.3,11 Manuscripts that do not meet these criteria should be considered for conditional acceptance (pending revisions) or rejection.3 Manuscripts with only minor flaws should be recommended for conditional acceptance (accept with revisions or accept with minor revisions) with the understanding that the reviewer’s suggestions for improving the manuscript are discretionary and may ultimately be ignored by authors.4,10 In contrast, manuscripts with major flaws should be recommended for conditional acceptance (accept with revisions or accept with major revisions) only if the authors satisfactorily address the reviewers’ concerns.1,4,10 When deliberating their publication recommendation, reviewers should give priority to innovative works that may have methodological flaws (minor or major) over manuscripts that are merely confirmatory or likely to be of low interest to the journal’s readership.4 The use of research methods that are unethical, do not follow contemporary norms (e.g., outdated or discredited methods), or cannot be replicated is grounds for recommending the rejection of a manuscript.1,11,13 Conclusion. The quality of peer reviews varies greatly and is often misguided, resulting in challenges for journal editors and frustrations for authors. The use of a stepwise approach when conducting a peer review can improve the quality of reviews, ultimately leading to high-quality articles for publication. Dr. Haines serves as a scientific editor and receives an honorarium from Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. The authors have declared no other potential conflicts of interest. This commentary is based on a presentation delivered at the ASHP Midyear Clinical Meeting, Las Vegas, NV, December 4, 2016.