Coupling Human Action Recognizer with Object Detector for Robotic Manipulation in Human-Robot Interaction

Establishing a communication mechanism for service and assistive robots to help the disabled and old age population through Human-Robot Interaction (HRI) is still a challenging task for further study. This research explores a vision and gesture-based control strategy, taking into account static and dynamic human gestures and full-body pose estimation without the use of any wearable devices. The human action recognition algorithm is coupled with an object detector algorithm to automatically perform a robotic task. The Deep Learning (DL) techniques, MediaPipe with Long Short-Term Memory (LSTM) model and You Only Look Once (YOLO) v9 algorithms are implemented for human gesture recognition and object detection respectively. For the robotic manipulation task, there are eight object classes, each uniquely associated with one of eight corresponding human gesture classes. These gestures are categorized into two types: four static gestures and four dynamic gestures. The dataset of object and gesture classes is prepared from scratch in this study. Based on the particular human gesture or action, the Intel RealSense depth camera provides position and depth information of the detected object, and the robotic arm picks and places that object using its Inverse Kinematics (IK) solution. The prediction outcomes and experimental demonstrations suggest that the proposed study performs quite well as compared to the recently used related studies and exhibits remarkable performance in establishing a real-time HRI system.