Hybrid Zero-Shot Interactive Navigation with LLMs: Path Planning Under Dual Constraints of Speech and Environment

This study proposes a novel hybrid interactive navigation framework for mobile robots, designed to enable robots to operate under dual constraints imposed by both the environment and the speech of accompanying humans in challenging future collaborative working scenarios. By leveraging human perceptual capabilities, the proposed framework significantly enhances the obstacle-avoidance capabilities and flexibility of robots. Specifically, an innovative composition algorithm is introduced to integrate traditional costmap-based navigation with a newly proposed LLM-assisted voice-based interaction method, thereby achieving real-time human–robot collaborative navigation with complementary advantages. Within this framework, robots can not only rely on spatial sensors to avoid obstacles but also follow verbal instructions from humans to bypass hazards that are difficult to detect. Moreover, the volume of speech is innovatively incorporated as a fusion weight, allowing the accompanying human to naturally guide the robot through voice volume modulation. To validate the feasibility and performance of the proposed framework and algorithm, we conducted both simulation and real-world experiments. A series of ablation and comparative studies was conducted to evaluate the merits and limitations of various configurations, ultimately providing optimal configurations based on the results. This work expands the scope of real-time human–robot interaction in navigation, offering new perspectives for future research.