Ideas for judging merit in manuscripts and authors

The integrity and progress of any academic field depends on reliable measures for judging the merit of its publications and their authors. In ecology, as in other disciplines, analysis and debate concerning the merit evaluation process itself, including both its criteria and consequences, have become subjects of a growing body of literature (e.g. Lortie et al. 2007, Statzner and Resh 2010). Troubling sources of bias and controversy over the judgment of merit occur, impacting at least three levels of the dissemination process: journal merit, reflected most commonly by 'impact factor'; manuscript/paper merit, reflected in the peer-review and editor-decision processes for submitted manuscripts, and by citation metrics of published papers; and author merit, reflected in criteria associated with research grant competitions, and with job-application/promotion processes, which draw largely on outcomes connected with the first two levels. The consequences of these merit rankings and perceptions of 'success' are not trivial, scaling up with potentially broad implications — e.g. authors with higher merit scores secure greater funding and hence capacity for continued research, wider propagation of their ideas and discoveries, and thus greater potential impact on socio-culture/economic institutions and policy. Importantly, those who rise to the top may often do so via a variety of subjective approaches, contingencies, and partisanships — not just good science. The topic of merit judgment was also a conspicuous theme in several commentaries featured in our third (2010) volume of IEE. Supp and White (2010) highlight the effect of review articles in elevating a journal's impact factor, and call for implementation of separate metrics that distinguish this effect, thus facilitating more accurate between-journal comparisons of merit. Donaldson et al. (2010), and Hochberg (2010) revisit the 'tragedy of the reviewer commons' issue, with particular recommendation for alleviation through better use of qualified referees represented among graduate students, post-docs and junior researchers. Finally, Wardle (2010) presents results from analyses showing that the recently launched 'Faculty of 1000' model generally fails as a reliable measure for judging the merit of published articles. Serious challenges remain regarding the integrity and value of the conventional peer-review system, and its approach to assigning merit and vetting science. In our view, these challenges need to be met by the development , testing, and implementation of new tools — by practicing scientists, for scientists. As ecologists using journals, we must engage critically in the process and criteria for evaluating merit at all levels — or we …