Defining quality in peer review reports: a scoping review

Abstract This study examines the challenge of defining quality in peer-review reports, a crucial yet underexplored aspect of academic publishing. Reviewers are vital gatekeepers of scientific knowledge, but unclear skills and a lack of standardized guidelines have led to inconsistent and subjective practices, weakening the overall efficacy of the peer-review process. To address this issue, the primary objective of this paper is to answer the research question: How has literature addressed guidance for producing quality peer-review reports? A scoping review was conducted, utilizing Scopus, Web of Science, SpringerLink, ScienceDirect, PubMed, and SAGE databases to search for records using keywords related to guidelines for scientific peer reviewing. The review identified 111 primary studies offering recommendations on how to review scientific articles. Extracted data were analysed thematically, focusing on approaches to reviewing articles, manuscript evaluation criteria, and report-writing guidelines. The findings revealed six key categories of review criteria for evaluating scientific manuscripts: structural components, research approach, style, ethical conduct, scientific value, and overall suitability. Additionally, the review provides 70 actionable recommendations for writing peer-review reports and highlights eight essential quality features expected in review texts: constructive, specific, fair, thorough, courteous, consistent, objective, and readable feedback. This study contributes to developing a standardized guide for scientific reviewing, with a particular emphasis on supporting early-career reviewers. The findings encourage academic publishers, journal editors, and professional organizations to adopt the proposed guidelines to enhance consistency, reduce bias, and improve the peer-review process. They also provide a foundation for developing new tools to support the reviewing.