Answers are grounded in this paper and cite their sources.
As large language models (LLMs) are increasingly used to gather, process and disseminate scientific information, concerns have grown about their identity-related biases. However, national-background biases in scientific evaluation remain insufficiently understood. Existing studies are limited to a narrow range of countries, abbreviated materials and/or simplified evaluation criteria, and have reported inconsistent findings. Moreover, differences in the textual evaluations generated by LLMs have seldom been quantified. Here, we used four popular LLMs to evaluate 292 full-text manuscripts from 28 journals across 8 research fields under 12 assigned national backgrounds, yielding 14,016 manuscript assessments, each covering 28 evaluative dimensions. Further comment-based rescoring tests were conducted to quantify differences in their textual evaluations. The results showed systematic differences in both scores and comments despite unchanged scientific content, with the magnitude and statistical significance these differences varying across national backgrounds. Under different national backgrounds, LLMs could offer different views on the same issue, and give different scores alongside similar textual evaluation. Researchers and decision-makers should be aware of these biases associated with source identity, including national background, when using LLMs for information gathering and processing.
No takes yet. Share an insight, caveat, or question.
Kong et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: