To address the issue that existing evaluation benchmarks are hardly adaptable to the dialogue scenarios of game virtual characters, this study proposes the CharacterBench-Game evaluation framework based on CharacterBench. It adds the "Gameplay Service" and "Worldview Constraint" modules, and introduces the taskgoal parameter to optimize the evaluation of task objectives, thereby providing a quantifiable tool for the customization ability of Large Language Models (LLMs) for game IP characters. Experiments were conducted on the core characters of 4 games, namely The Legend of Zelda: Breath of the Wild, Detroit: Become Human, Cyberpunk 2077, and Tomb Raider. A 5-point scale was used to quantitatively evaluate dialogue scenarios, including both English and Chinese versions, to test GPT-4o and deepseek-llm-7b-chat. The results show that the framework can effectively distinguish the capabilities of different models: GPT-4o has an average score of 4. 1278 in Chinese and 3. 9389 in English, achieving full marks in the dimensions of memory and boundary consistency; deepseek-llm-7b-chat scores lower in all dimensions, with a significant gap in the Chinese dimension of factual accuracy. The newly added dimensions increase the evaluation accuracy from 68% to 89%, and the taskgoal parameter improves the task completion rate of open-source models by 22. 92%, which is 20. 5% better than that of closed-source models. This framework provides game industry with an evaluation tool for LLM character customization and a basis for model selection.
Xin Li (Wed,) studied this question.