Large Language Models in Psychology: Application in the Context of a Systematic Literature Review.

The present study assesses the potential of employing Large Language Models (LLMs) in the context of a systematic literature review in psychology. We tasked one of the currently available ChatGPT-4-turbo-preview models from OpenAI with a qualitative coding assignment, which involved identifying elements related to a specific theoretical-analytical framework within 39 scientific empirical papers. We evaluated the quality of LLM-generated outcomes by comparing them with results generated through traditional human coding. In the process, we outlined the capabilities and advantages of using LLMs for systematic literature reviews, including practical considerations for their implementation. Our analyses showed that the LLM produced results that aligned closely with those obtained through traditional human coding. Furthermore, our experience indicated that incorporating LLMs into our research workflow was time- and cost-effective. Our results suggest that researchers and LLMs can work synergistically, improving efficiency, cost-effectiveness, and quality of the systematic literature review process. We underline the critical role of human arbitration in prompt crafting and decision-making.