A Vision-based Correction of Inertial Measurement of Human Motion for Robot Programming by Demonstration

—We propose in this work an original approach consisting in correcting inertial human hand trajectory with vision-based object tracking in a context of programming by demonstration (PbD) of pick-and-place tasks. One challenge in PbD is to record human demonstrations accurately enough, in an easy way which does not limit human motion. Merging inertial-based and vision-based technologies may take advantage of both and fulfill the requirement of a PbD process. Our method is based on the identification of Positions of Interest (POIs) from object and hand data, corresponding to picking or placing actions. Then objects POIs are paired with hand POIs to modify the human hand trajectory. The method is implemented on a Sawyer robot with Xsens IMU sensors. Pick-and-place tasks with different complexity have been recorded and reproduced by the robot. The robot succeeds to reproduce the demonstrated tasks which validates our method.