Assessing Frontier LLMs in Solving Game Development Problems: Preliminary Findings Across Three Game Engines

This paper evaluates three frontier LLMs (ChatGPT-4o, o3, and Gemini 2.5 Pro) against expert-rated human answers for 30 technical questions collected from online Q&A forums about three popular game engines: Unreal, Unity, and Godot. Our results reveal significant performance variance, with o3 demonstrating superior capabilities over Gemini 2.5 Pro and ChatGPT-4o. A primary weakness identified across all models was response completeness, where AI-generated answers often lacked the comprehensive detail of the human baseline. These findings suggest that although LLMs are powerful assistants, they are not yet a substitute for human expertise in engine-based game development tasks.