Using dVRK teleoperation to facilitate deep learning of automation tasks for an industrial robot

Deep Learning from Demonstrations (Deep LfD) is a promising approach for robots to perform precise bilateral automation tasks involving contact and deformation, where dynamics are difficult to model explicitly. Deep LfD methods typically use datasets of 1) human videos, which do not match robot kinematics and capabilities or 2) waypoints collected with tedious move-and-record interfaces, such as teaching pendants or kinesthetic teaching. We explore an alternative using the Intuitive Surgical da Vinci, which combines a pair of gravity-balanced, high-precision, passive, and 6-DOF master arms with stereo vision, allowing humans to teleoperate precise surgical automation tasks. We present DY-Teleop, an interface between the da Vinci master manipulators and an ABB YuMi industrial robot to facilitate the collection of time-synchronized images and robot states for deep learning of automation tasks involving deformation and dynamic contact. The system has an average latency of 194ms and executes commands at 6Hz. We present YuMiPy, an open source library and ROS package for controlling an ABB YuMi over Ethernet. Data collection experiments with scooping a ball into a cup, untying a knot in a rope, and pipetting liquid between two containers suggest that demonstrations obtained by DY-Teleop are comparable with those by kinesthetic teaching in demonstration time. We performed Deep LfD for the scooping task and found that the policy trained with DY-Teleop achieved a 1.8× higher success rate than a policy trained with kinesthetic teaching. Code, videos, and data are available at berkeleyautomation.github.io/teleop.