Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMs

When creating plans, policies, or applications for people, it is challenging for designers to think through the strategic ways that different people will behave. Recently, Large Language Models (LLMs) have been shown to create realistic simulations of human-like behavior based on personas. We build on this to investigate whether LLMs can simulate human strategic behavior. Human strategies are complex because they take into account social norms in addition to aiming to maximize personal gain. The ultimatum game is a classic economics experiment used to understand human strategic behavior in a social setting. It shows that people will often choose to "punish" other players to enforce social norms rather than to maximize personal profits. We test whether LLMs can replicate this complex behavior in simulations. We compare two architectures: single- and multi-agent LLMs. We compare their abilities to (1) simulate human-like actions in the ultimatum game, (2) simulate two player personalities, greedy and fair, and (3) create robust strategies that are logically complete and consistent with personality. Our evaluation shows the multi-agent architecture is much more accurate than single LLMs (88% vs. 50%) in simulating human strategy creation and actions for personality pairs. Thus there is potential to use LLMs to simulate human strategic behavior to help designers, planners, and policymakers perform preliminary exploration of how people behave in systems.