What question did this study set out to answer?

The aim is to enhance zero-shot generalization in multi-agent systems by leveraging large language models to generate diverse strategies.

March 10, 2026Open Access

LLM-TOC: LLM-Driven Theory-of-Mind Adversarial Curriculum for Multi-Agent Generalization

Key Points

The aim is to enhance zero-shot generalization in multi-agent systems by leveraging large language models to generate diverse strategies.
Introduced LLM-TOC as a bi-level Stackelberg game model for strategy generation.
Developed Gradient Saliency Feedback to guide the language model in discrete code synthesis.
Applied PAC-Bayes framework for theoretical analysis of convergence and generalization error.
Tested the model against baselines in multi-agent reinforcement learning environments.
LLM-TOC achieved 75% to 85% of the performance of the top-tier Oracle PPO.
Outperformed self-play baselines (IPPO and MAPPO) across all test setups.
Reduced total training computational cost by over 60% compared to mainstream methods.

Abstract

Zero-shot generalization to out-of-distribution (OOD) teammates and opponents in multi-agent systems (MASs) remains a fundamental challenge for general-purpose AI, especially in open-ended interaction scenarios. Existing multi-agent reinforcement learning (MARL) paradigms, such as self-play and population-based training, often collapse to a limited subset of Nash equilibria, leaving agents brittle when faced with semantically diverse, unseen behaviors. Recent approaches that invoke Large Language Models (LLMs) at run time can improve adaptability but introduce substantial latency and can become less reliable as task horizons grow; in contrast, LLM-assisted reward-shaping methods remain constrained by the inefficiency of the inner reinforcement-learning loop. To address these limitations, we propose LLM-TOC (LLM-Driven Theory-of-Mind Adversarial Curriculum), which casts generalization as a bi-level Stackelberg game: in the inner loop, a MARL agent (the follower) minimizes regret against a fixed population, while in the outer loop, an LLM serves as a semantic oracle that generates executable adversarial or cooperative strategies in a Turing-complete code space to maximize the agent’s regret. To cope with the absence of gradients in discrete code generation, we introduce Gradient Saliency Feedback, which transforms pixel-level value fluctuations into semantically meaningful causal cues to steer the LLM toward targeted strategy synthesis. We further provide motivating theoretical analysis via the PAC-Bayes framework, showing that LLM-TOC converges at rate O(1/K) and yields a tighter generalization error bound than parameter-space exploration under reasonable preconditions. Experiments on the Melting Pot benchmark demonstrate that, with expected cumulative collective return as the core zero-shot generalization metric, LLM-TOC consistently outperforms self-play baselines (IPPO and MAPPO) and the LLM-inference method Hypothetical Minds across all held-out test scenarios, reaching 75% to 85% of the upper-bound performance of Oracle PPO. Meanwhile, with the number of RL environment interaction steps to reach the target relative performance as the core efficiency metric, our framework reduces the total training computational cost by more than 60% compared with mainstream baselines.

Read Full Paperexternally

Perguntar à IA

Bookmark

View Full Paper