Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables

We develop an optical multimodal-sensing skin for fingers of a robotic gripper. This sensor consists of a transparent skin made with a marked soft elastic outer layer on a hard layer, and internal RGB cameras. The cameras can see the external scene through the skin. This sensor provides a contact force field estimated by tracking markers in the skin, and visual information of close or grasped objects. This sensor uses off-the-shelf materials, is easy to manufacture, and is robust against an external force as the force is not applied to the sensing devices (cameras). We install a prototype of this sensor on fingers of a Baxter robot, and demonstrate its usefulness with a cutting vegetable task. The sensor gave us force information to control cutting vegetables and avoiding slippage of the knife. Accompanying video: https://youtu.be/ifOwQdy9gDg.