DyGRAV: Dynamic Grounding and Relational Reasoning for Vision–Language Navigation

Vision–Language Navigation (VLN) requires embodied agents to interpret natural-language instructions in 3D environments. While significant progress has been made, state-of-the-art agents often struggle with instructions that demand fine-grained visual understanding, correctly grounding specific object attributes, distinguishing between similar objects, and interpreting complex spatial relationships. Many methods rely on static visual features or on global cross-modal attention and lack mechanisms to dynamically resolve ambiguities during navigation. We propose DyGRAV (Dynamic Grounding and Relational Reasoning for VLN), which is a dynamic, uncertainty-aware framework that augments a strong VLN backbone with two adaptively triggered reasoning modules. The Contextual VLM Querying module detects grounding ambiguity for descriptive phrases (e.g., “the ceramic bowl”) and issues targeted queries to a vision–language model (e.g., CLIP) using current visual evidence, while the Incremental Relational Scene Graph (IRSG) constructs lightweight, task-relevant relational memory online to resolve complex spatial relations (e.g., “the chair next to the window”). A budget-aware gating mechanism regulates when these modules are invoked, optimizing the balance between reasoning quality and computational cost through value-per-cost selection and hard budget constraints. Comprehensive evaluations on R2R and RxR, along with a fine-grained diagnostic benchmark emphasizing attribute and relation understanding, show that DyGRAV achieves superior navigation accuracy on challenging instructions while maintaining bounded per-step compute budgets with 97.33% episode-level adherence.