A Simulation-Based VQA Dataset for Evaluating Intrinsic Physical Property Inference Capabilities of Vision-Language Models

Key Points

• The aim is to create a benchmark to evaluate how well vision-language models can infer physical properties like mass and elasticity through interactions in simulated environments.
• Developed a multi-view video question-answering benchmark using simulation environments with inverse kinematics control.
• Designed tasks like cube pushing and sphere dropping to collect data.
• Analyzed performance by fine-tuning state-of-the-art models on the dataset.
• Pre-trained models showed low accuracy; fine-tuning significantly improved performance on static mass inference tasks.
• In dynamic elasticity inference tasks, performance improvements were limited, showing a return to verbal bias.
• The dataset provides a quantitative assessment environment for physical reasoning capabilities in models.