Robot-Manipulator Teleoperation by Markerless Vision-Based Hand-Arm Tracking

This article presents a method of real-time robot-manipulator teleoperation using markerless image-based hand-arm tracking. The markerless tracking is carried out by processing images from two calibrated cameras to estimate in three dimensions (3-D), the positions of the wrist joint, elbow joint, index finger, and thumb. The hand pose (position and orientation) is used to specify the pose of the end-effector of a robot-manipulator in real-time teleoperation. The method permits a natural means of communicating an entire task to a robot, rather than using limited motion commands as with gesture-based approaches, and it has been demonstrated for pick-and-place tasks.