Peer review: Least-worst approach or the very best we can do?

The Editorial Board of the Australian Occupational Therapy Journal uses a review process to determine suitability of submissions for publication. An important part of this process is peer review. Peer reviewers provide independent advice to the Editorial Board regarding the scholarly merit of a submission. Although ‘in the end it is the journal editor who accepts or rejects an article’ (Occupational Therapy Australia, 2011), the advice of peer reviews is ‘pivotal’ (MacKenzie, 2006). An aspiring author sends a manuscript to a journal's editorial office. The journal editor, or for a large journal one of the associate editors, logs in the manuscript, selects two or three referees to evaluate the manuscript, and sends each a copy. Reviewers are asked to assess the manuscript and make a recommendation to accept, accept with revisions, or reject the manuscript. The editor or associate editors then decide if they will accept the recommendation of the reviewers. If the reviewers disagree, the editor may subject the manuscript to another round of reviews, or the editor may adjudicate, deciding without further review if the manuscript should be accepted or rejected. Within these standard parameters, many variations exist. (Weller, 2001, p. 2) The Australian Occupational Therapy Journal and its predecessor the Bulletin, have used variations of this process for over 70 years. The description fits current practice of the journal with the exception of manuscript receipt, which is done through a web-based platform Scholar-one™ enabled and supported by the international publishing company WILEY which is contracted by OT Australia to produce the journal. Apart from receipt, tracking and production functions, every other step is performed by OT Australia members who are appointed to the Editorial Board in honorary positions or serve as volunteer reviewers. Peer reviewers are key contributors to the quality of a journal (MacKenzie, 2006). They have been described as the ‘lynchpin about which the whole business of Science is pivoted’ (Ziman, 1968, p. 111, cited by Weller, 2001). Their acceptance of the task, the timeliness and calibre of their reports and recommendations underpin journal quality and thus the knowledge base of the discipline. Many hundreds of people have contributed to Australian Occupational Therapy Journal quality through voluntary review of submissions. Fricke (2007), for example, identified over 140 people had reviewed between one and four manuscripts within an 18-month period of Australian Occupational therapy Journal production. Peer reviewers should thus have necessary content, methodological and theoretical expertise to make informed judgements about submissions. Individual expertise is what gives the process validity because inevitably reviewer decisions are subjective. One reviewer may commend a submission on its originality, whereas another may say it adds nothing new. One reviewer may endorse a particular methodology, whereas another may say the same study has fatal procedural flaws. The subjective nature of review and editorial decisions has been identified as one of a number of problems of peer review. Peer review is inherently time consuming, uncertain and complex. As Professor Sir Brian Heap, Chair of the Sense About Science Working Party on Equipping the Public with an Understanding of Peer Review said, ‘few outside the scientific world can be expected to know about the immense effort, and even pain, that is experienced before research is ready to be published as a peer-reviewed paper in a scientific journal. The peer review process is almost as complicated and unpredictable as the birth of a baby. Yet, as with delivery, it involves analysis, judgement and evaluation … things can go wrong’ (2004, vii). Research into the editorial process and peer review has tended to focus on what can go wrong. Peer review research began in the mid-20th Century (Weller, 2001), revealing a myriad of problems. Some of these are experienced from time to time by the Australian Occupational Therapy Journal. Most relate to practicalities of the process itself. They include: difficulty in defining what, for a particular submission, a ‘peer’ might be; the time it takes to locate potential reviewers; delays between invitation and reviewer agreement or decline; difficulty in obtaining agreements to review from already busy, in demand and over-burdened experts; reviewers agreeing then not completing reports; reviewers needing adequate time for review; reviewers requiring time extensions; reviewers realising after starting the review that they do not have expertise at the level they think is required; variability in the level of detail, methodological or theoretical rigor of reviews; the potential for theoretical or methodological bias; and the reality that even well-meaning, expert and diligent reviewers can make mistakes. There is a lot that can go wrong in journal editorial processes that involve peer review. Indeed a former editor of the British Medical Journal has said peer review is ‘ineffective, largely a lottery, anti-innovatory, slow, expensive, wasteful of scientific time, inefficient, easily abused, prone to bias, unable to detect fraud and irrelevant’ (Smith, 2015). So why does it remain the gold-standard quality assurance strategy in scholarly publications? Why does the Australian Occupational Therapy Journal use it? Professional society journal publication without peer review, or something like it, is unthinkable. The difference between scholarly journal publications and media announcements, news, direct release of research and advertising is that ‘peer review of a scientific paper involves the addition of qualifications and limitations on conclusions’ (Heap et al., 2004; p. vii). These caveats mean the implications of research are considered in context, with appropriate caution regarding consequences and potential applications. The public does not have the depth or breadth of knowledge to do this for the vast range of papers that are published. Peer review uses the expertise of people who have a deep understanding of the problem, theory or method being reported to help determine whether the work is competent and importantly whether author conclusions are plausible. A study may be procedurally correct, but that does not necessarily mean author claims are justified. An expert can evaluate claims in light of research evidence as reported. They can evaluate the conceptual foundations and conclusions of theoretical work. It is only peers who can adequately interrogate the credibility of conclusions: ‘any original piece of research has its own unique features, requiring the application of expert judgement in ways that no one yet has been able to codify in a check-list. It would be no easier to devise a checklist to replace referees’ judgements than it would be to prepare one to replace medical diagnoses’ (Research Information Network, 2010, p. 18). Yes there are risks to innovation in a process that relies on existing expertise and current perspectives, but perhaps this offers less potential harm than a dissemination approach where there is no value added through cautious evaluation. So, although peer review is cumbersome, time consuming, resource intensive and vulnerable to human error, weakness or maleficence, it is a ‘system full of problems but the least worst we have’ (Smith, 2006, p. 178). The Australian Occupational Therapy Journal uses peer review as one of a number of strategies to assure the public that we are doing what we can to present work that peers consider is competent and credible with conclusions that are responsible and plausible. How has the Australian Occupational Therapy Journal made the ‘least-worst’ system the best it can be? Over many decades, the editorial review process of the journal has been considered, critiqued and changed (e.g. Australian Association of Occupational Therapists, 1999; Cusick, 1988, 1990; Fricke, 2007; Froude, 2010, 2014; MacKenzie, 2006). Innovations have been introduced consistent with: emerging research about strategies to enhance the efficiency and effectiveness of peer review; technological advances in manuscript receipt, reviewer databases and monitoring; enhanced editorial experience; author and reviewer feedback; regulatory, research and professional association standards; publishing company policies and practices. As the number of submissions to the Australian Occupational Therapy Journal increases (Fricke, 2007; Froude, 2014), the demand for peer review expands. The Editorial Board needs to make sure that we adopt strategies to meet this demand at the same time as maintaining quality. There are many examples of quality improvements made over time to the Australian Occupational Therapy Journal peer review process: establishing a reviewer database and adapting/upgrading database platforms with technological advances; producing a manual for reviewers/referees; introducing blind review; introducing reviewer feedback by each with a de-identified copy of the other's report and the decision and comments of the Editor; introduction of standardised reviewer letters, feedback and rating forms; publishing the names of people who have reviewed each year; and personally thanking reviewers at hosted social events held in conjunction with OT Australia national conferences. These strategies have been important innovations, but as more information and research evidence about strategies to enhance peer review becomes available more change is needed. In 2016, the Australian Occupational Therapy Journal will therefore amend some of the ways it manages peer review to build on the past and invest in the future. Three projects have been identified. First, streamlining the reviewer database following a 2015 audit of the Australian Occupational Therapy Journal Scholar-one archive; the results of that project are reported here along with the 2016 next step (Figure 1). Second, a Review Board will be established in late 2016, similar to those used in many other biomedical and health journals. This will better acknowledge high volume, high quality reviewer contributions and increase the consistency of review standards. Third, a survey of reviewers will occur ‘10 years on’ from the last one in 2006 (MacKenzie, 2006). Reviewers said then that mentoring, provision of feedback and the opportunity to attend courses about reviewing was something they wanted to see developed. Professional development opportunities will be prepared consistent with the 2016 findings. In 2006 Associate Editor MacKenzie surveyed N = 178 Australian Occupational Therapy Journal reviewers. In 2015, the Scholar-one database revealed over 2000 entries. What happened? Back in 2006 the first online manuscript submission and review system had been implemented through Manuscript-Central. Automated data collection regarding authors and reviewers was possible for the first time leading to an expansion in the journal database. It appears that ‘reviewer’ was included as a default code for all Scholar-one log-ins when the new system was adopted. As a result, all corresponding authors were identified as potential reviewers creating a vast data warehouse where finding suitable reviewers could be time consuming and challenging. The 2015 audit was conducted by Anne Cusick, the Editor in Chief, with data retrieval and the first round of descriptive analysis performed by WILEY publishing staff Alison Bell and Meg A'Hearn. The results of the audit are presented in Figure 1. All n = 535 active reviewers will be surveyed in 2016 to seek their consent to remain on the contact list, find out more about their reviewer experience, attributes and professional development and review support needs. The Editor in Chief will then approve all consenting and new reviewers for agreed terms. Having accurate, up-to-date information about reviewers is one of a number of strategies to enhance the efficiency and effectiveness of peer review. A number of initiatives relating to peer review in the Australian Occupational Therapy Journal will be progressed in 2016: database streamlining, reviewer survey and professional development opportunities, and launch of a Review Board to work in conjunction with Scholar-one™ and guest reviewers. If peer review is the least-worst system we have, we need to make sure that we do everything possible to make it the very best we can do.