Evaluating the Viability of Foundation Models in Virtual Agent Perception

This work presents a comparative study evaluating how recent foundation models, capable of handling various computer vision tasks, perform at simulating a virtual agent's perception and using that perception to navigate the environment and achieve its goals. To achieve this, we selected two recent foundation models well known in the literature and compared them with a recent procedural approach by replicating these models' evaluation methods. We employed these models for evacuating a burning building, aiming to identify either exit signs or people to follow in the environment to find an escape route. Our results show that out‐of‐the‐box foundation models were unable to outperform a specialized procedural method and performed significantly worse on the task, although they still exhibit strong adaptability to previously unknown contexts. However, among the two foundation models tested, we found that a VLM capable of image reasoning and detection performed significantly better than a self‐supervised generalized computer vision model.