Abstract Large language models (LLMs) are increasingly used for persuasion, such as in political communication and marketing. Yet, empirical findings on the effectiveness of LLMs in persuasion compared to humans remain inconsistent. Here, we conduct a systematic literature review and meta-analysis to quantify the persuasion effect of LLMs compared to humans. We identified 8 studies with 20, 184 participants and 12 effect size estimates. We then compute the standardized effect sizes based on Hedges’ g. The results show no significant overall difference in persuasive performance between LLMs and humans (g = 0.01, p = 0.773). However, we observe substantial heterogeneity across studies (I2 = 64.78%), suggesting that persuasiveness strongly depends on contextual factors. In separate moderator analyses, no individual factor (e.g., LLM model, interaction type, or domain) reached statistical significance, which may be due to the limited number of studies. When considered jointly in a combined model, these factors explained a large proportion of the between-study variance (R2 = 83.95%), and residual heterogeneity is low (I2 = 14.30%). This suggests that differences in LLM model, interaction type, and domain interact in shaping persuasive performance, and that single-factor tests may understate their influence. Our results highlight that LLMs can match human performance in persuasion, but their success depends strongly on how they are implemented and embedded in communication contexts.
Hölbling et al. (Mon,) studied this question.