Evaluating Large Language Models in Psychological Research: A Guide for Reviewers

Large Language Models (LLMs) are being increasingly used in scientific research, be it to analyze data, generate synthetic data, or even to write scientific papers. This trend necessitates that journal reviewers are able to evaluate the quality of works that utilize LLMs. We provide reviewers of psychological research with a comprehensive guide on evaluating research that uses LLMs, examining their dual roles of automating data processing and simulating human data. Essential considerations for reviewers are highlighted, focusing on the evaluation of methodological rigor, the importance of replicability, and the validity of results when employing LLMs. We offer practical advice on assessing the appropriateness of LLM applications in submitted studies, emphasizing the need for transparency in methodological reporting and the challenges posed by the non-deterministic and continuously evolving nature of these models. By providing a framework for critical review, this guide aims to ensure high-quality, innovative research within the evolving landscape of psychological studies utilizing LLMs.