Answers are grounded in this paper and cite their sources.
Benchmark evaluates grounding and reasoning capabilities of LLMs in strategic Pokémon battles, indicating gaps in existing game knowledge.
Developing grounding techniques for LLMs poses two requirements for interactive environments, i.e. , (i) the presence of rich knowledge beyond the scope of existing LLMs and (ii) the complexity of tasks that require strategic reasoning. Existing environments fail to meet both requirements due to their simplicity or reliance on commonsense knowledge already encoded in LLMs for interaction. In this paper, we present PokéLLMon, a new benchmark enriched with fictional game knowledge and characterized by the intense, dynamic, and adversarial gameplay of Pokémon battles, setting new challenges for the development of grounding and reasoning techniques in interactive environments. Empirical evaluations demonstrate that existing LLMs lack game knowledge and struggle in Pokémon battles. We investigate grounding techniques that leverage feedback and game knowledge, and provide a thorough analysis of reasoning methods from a new perspective of action consistency. Additionally, we introduce higher-level reasoning challenges when playing against human players. The implementation of our benchmark is released at: https://github.com/git-disl/PokeLLMon.
No takes yet. Share an insight, caveat, or question.
Hu et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: