Abstract This study proposes a conformity score metric to facilitate the editorial evaluation of peer review report quality. This metric assesses review reports’ conformity with quality standards by identifying and quantifying the contribution of individual sentences within the content. When the conformity score metric is applied to a review report, sentences relating to the methodological aspects of the research reviewed are assigned the highest score relative to sentences relating to the reviewers’ discussions of other aspects of the reviewed research. To identify review sentences focusing on different aspects of the reviewed research, this study uses the Long Short-Term Memory (LSTM) model, which classifies sentences from peer review report texts into a number of categories. Based on review report textual data, this study examines the relationship between features that are commonly regarded as indicators of review report quality and conformity scores, which are determined to have a strong correlation with these textual features. Overall, this study proposes a review report quality assessment metric that considers the semantic meaning of review content and emphasizes discussions about the scientific aspects of the reviewed research. The conformity score metric and the LSTM model can thus be integrated into review report examination practices.
Sun et al. (Thu,) studied this question.