VIG-Un: An Interactive Vision and Instruction Grasping for Unknown Objects

Despite the development of various dexterous grasping techniques, it remains challenging for robots to successfully pick unknown objects through an image or speech input provided by a human. The challenge lies in the absence of an effective mechanism to integrate visual information and instruction derived from human interaction, and to subsequently associate this integration with unknown objects captured by sensors in a real-world scenario. To address this challenge, this work proposes an interactive vision and instruction grasping method for unknown objects (VIG-Un). The proposed VIG-Un offers users three communication modes: visual interaction, instruction interaction, and vision–instruction combined interaction. In the vision-based mode, ARUCO marker boards serve as input samples for object detection, providing a reliable visual reference for accurate object localization. In the instruction mode, we introduce a Semantic-Aware Image Generation Branch (SAIGB), which generates relevant images based on textual instructions, enabling semantic understanding in grasping tasks. Then, VIG-Un leverages a Deep Siamese Network (DSN)-based few-shot object detection approach, reducing the reliance on visual fine-tuning and substantially enhancing the flexibility and adaptability of robotic manipulators during grasp operations. In experimental evaluations benchmarked against state-of-the-art methods, including YOLO-World and RT-1, the VIG-Un framework achieves higher object detection accuracy and grasping success rates than these competitors on object-grasping tasks in typical indoor environments, while maintaining acceptable execution efficiency, demonstrating its effectiveness and robustness.