Hybrid Visual Perception and LLM Framework for Service Robot Planning and Navigation

In the last decade, service robots have gained significant attention for their role in supporting humans across various applications. To operate effectively and adaptively, service robots require environmental perception, natural-language understanding, and planning and navigation capabilities. However, traditional planning frameworks struggle to provide the required flexibility. In contrast, end-to-end multimodal systems based on vision-language models (VLMs) require substantial computational resources, making them unsuitable for deployment on embedded robotic platforms. This paper proposes a hybrid framework that combines fast, deep-learning-based visual perception with the reasoning capabilities of a large language model (LLM) to enhance task comprehension, planning, and navigation for service robots. By decoupling vision from language reasoning, we avoid the complexity of a unified vision-language model, allowing the proposed framework to be deployed on a lightweight embedded platform with substantially lower latency than the VLM baseline. Experimental results show a 96\% task completion rate in structured mission tests and 84\% overall success in hybrid perception tasks. Moreover, the proposed approach achieves 3× to 20× faster inference than VLM-based baselines under both API-based and local deployment settings. Real-robot deployment confirmed the successful transfer of the full perception-to-action pipeline to the physical platform.