Robotic bolt insertion and tightening based on in-hand object localization and force sensing

To perform a complex manipulation task, we need to know the position and posture of the object in hand, that is, in-hand object localization, and measure a force applied to the grasped object. We propose multimodal tactile sensing with an optical device that can perform in-hand object localization and force sensing. We applied this sensor to a manipulation task, inserting a bolt into the target hole and tightening the bolt, based on these tactile information. The proposed tactile sensor uses a combination of a reflective membrane-based method that uses an elastomer sheet coated with reflective pigments and a marker displacement-based method to obtain tactile information through a camera. In-hand object localization is done by obtaining detailed surface texture as a shading image reflecting a deformation of the elastomer surface. Force is measured by detecting the displacement of markers embedded in an elastomer. Through experiments with a 6 DoF robotic arm equipped with the proposed sensor, we show that our approach enables a screw inserting and temporally tightening task using only information from the proposed sensor.