• Whether peer review quality significantly associates with citations is investigated. • Peer review quality is measured using a large language model (Gemini 2.0 Flash) based on a well-developed review quality instrument to avoid subjective and time-consuming concerns. • The comparison between Gemini 2.0 Flash and human evaluators demonstrates good and excellent agreement. • Linear regression results and several robustness checks indicate a positive association between peer review quality and citations. • The widespread utilization of large language models still warrants caution despite the present promise. The significant rise in scientific publications has made pre-publication peer review crucial for safeguarding research quality and enhancing scientific impact. Yet, correlation between peer review quality, indicative of a rigorous peer review procedure, and the subsequent scientific impact is rarely investigated. Here, this study leverages Google’s large language model (LLM) named Gemini 2.0 Flash along with the few-shot prompt to automatically evaluate peer review quality, and investigate the correlation between the automatically assessed peer review quality (indicated by review scores) and scientific impact (measured by post-publication citations). First, the subsample reliability analysis reveals excellent agreement between review scores evaluated by humans and those generated by Gemini 2.0 Flash, underscoring the potential of LLMs in automatically assessing peer review quality. Then, empirical analysis indicates a positive association between automatically assessed review scores and citations. Specifically, a one-point increase in the review score is associated with an average increase of approximately five citations (coefficient = 5.440, p < 0.01). This positive effect remains robust in several robustness analyses, including transforming citation counts into logarithmic form, excluding papers within the top 1% of citations, and applying a negative binomial regression model. In addition, all analyses remain robust when using the average and “year-normalised” citations as dependent variables, when using first-author information to proxy control variables, and when employing a zero-shot prompt for Gemini 2.0 Flash responses, despite the relatively lower estimated effect. These findings highlight the importance of peer review quality in enhancing the scientific impact and the potential of using LLMs to automatically assess the review quality. Despite the promise, the widespread adoption of LLMs in automatically assessing peer review relevant tasks warrants further investigation.
Zhuanlan Sun (Mon,) studied this question.