Learning-Finding-Giving: A Natural Vision-Speech-based Approach for Robots to Assist Humans in Human-Robot Collaborative Manufacturing Contexts

Human-robot collaboration can improve and enhance current manufacturing processes, in which robots are able to provide collaborative assistance to humans, allowing for increased productivity and minimal time waste. The use of everyday mechanical appliances and tools is unavoidable, making these handover tasks common in human-robot collaborative manufacturing contexts. A typical handover task can be performed in three general steps: object identification, object grasping, and object handover. In this work, we propose a learning-finding-giving framework based on computer vision and speech recognition approaches for robots to dynamically identify and deliver tools for human partners in collaborative tasks. The YOLOv5 object detection algorithm is utilized for the identification of common mechanical tools. To teach robots to understand the target objects, a custom dataset is created from over 2000 images of the mechanical tools, followed by the implementation in real-world human-robot collaborative tasks. Experimental results and evaluations show that the proposed solution allows robots to dynamically understand and grasp tools with high accuracy, effectively assisting in handover tasks for human teammates.