CharacterBench-Game: A Quantitative Evaluation Framework for Large Language Models in Game Fictional IP Character Dialogue

To address the issue that existing evaluation benchmarks are hardly adaptable to the dialogue scenarios of game virtual characters, this study proposes the CharacterBench-Game evaluation framework based on CharacterBench. It adds the "Gameplay Service" and "Worldview Constraint" modules, and introduces the task_goal parameter to optimize the evaluation of task objectives, thereby providing a quantifiable tool for the customization ability of Large Language Models (LLMs) for game IP characters. Experiments were conducted on the core characters of 4 games, namely The Legend of Zelda: Breath of the Wild, Detroit: Become Human, Cyberpunk 2077, and Tomb Raider. A 5-point scale was used to quantitatively evaluate dialogue scenarios, including both English and Chinese versions, to test GPT-4o and deepseek-llm-7b-chat. The results show that the framework can effectively distinguish the capabilities of different models: GPT-4o has an average score of 4.1278 in Chinese and 3.9389 in English, achieving full marks in the dimensions of memory and boundary consistency; deepseek-llm-7b-chat scores lower in all dimensions, with a significant gap in the Chinese dimension of factual accuracy. The newly added dimensions increase the evaluation accuracy from 68% to 89%, and the task_goal parameter improves the task completion rate of open-source models by 22.92%, which is 20.5% better than that of closed-source models. This framework provides game industry with an evaluation tool for LLM character customization and a basis for model selection.