Empowering Robots with Multimodal Language Models for Task Planning with Interaction

This paper introduces a multimodal language model based agent system for task planning with interaction integrated into a physical robot to facilitate enhanced human-robot interactions. The system architecture integrates textual and visual inputs directly within the system, allowing for seamless transitions between conversational and task-oriented interactions. The agent system comprises a router, a chatbot, and a task planner enabling efficient decision-making and flexibility in real-world applications. We conduct in-depth study with text-only interaction and multimodal interaction, our extensive experiments find that multimodal interaction receives higher score by human evaluation and improves success rates in task planning. Our system simplifies the architecture by incorporating multiple agents for specific tasks while improving both the system's efficiency and cost-effectiveness. The robot's ability to interact using multimodal inputs significantly enhances user experience, as evidenced by evaluations measuring friendliness, usefulness, and task completeness. Practical implications are vast, promising advancements in robotics applications for personal and professional environments.