Robot grasp detection using multimodal deep convolutional neural networks

Autonomous manipulation has enabled a wide range of exciting robot tasks. However, perceiving outside environment is still a challenging problem in the field of intelligent robotic research due to the lack of object models, unstructured environments, and time-consuming computation. In this article, we present a novel robot grasp detection system that maps a pair of RGB-D images of novel objects to best grasping pose of a robotic gripper. First, we segment the graspable objects from the unstructured scene using the geometrical features of both the object and the robotic gripper. Then, a deep convolutional neural network is applied on these graspable objects, which aims to find the best graspable area for each object. In order to improve the efficiency in the detection system, we introduce a structured penalty term to optimize the connections between multimodality, which significantly mitigates complexity of the network and outperforms fully connected multimodal processing. We also present a two-stage closed-loop grasping candidate estimator to accelerate the searching efficiency of grasping-candidate generation. Moreover, the combination of a two-stage estimator with the grasping detection network naturally improves detection accuracy. Experiments have been conducted to validate the proposed methods. The results show that our method outperforms the state of the art and runs at real-time speed.