Evaluating the persuasive influence of political microtargeting with large language models

Recent advancements in large language models (LLMs) have raised the prospect of scalable, automated, and fine-grained political microtargeting on a scale previously unseen; however, the persuasive influence of microtargeting with LLMs remains unclear. Here, we build a custom web application capable of integrating self-reported demographic and political data into GPT-4 prompts in real-time, facilitating the live creation of unique messages tailored to persuade individual users on four political issues. We then deploy this application in a pre-registered randomized control experiment (n = 8,587) to investigate the extent to which access to individual-level data increases the persuasive influence of GPT-4. Our approach yields two key findings. First, messages generated by GPT-4 were broadly persuasive, in some cases increasing levels of support for an issue stance by nearly 50%. Second, in aggregate, the persuasive impact of microtargeted messages was not statistically different from that of non-microtargeted messages (5.68% vs 7.32%, respec- tively, P = 0.082). These trends hold even when manipulating the type and number of attributes used to tailor the message. Taken together, these findings suggest — contrary to widespread speculation — that the influence of current LLMs may reside not in their ability to tailor messages to individuals, but rather in the persuasiveness of their generic, non-targeted messages. This work secondarily contributes by offering a robust and replicable approach – through a custom web-based pipeline – to integrating LLMs into experimental designs, and a novel dataset, GPTarget2023, containing metadata for thousands of tailored AI-generated messages.