Knowledge-augmented embodied exploration for robotic grasping in constrained environments

It has always been expected that robots can actively manipulate complex environments to fulfill human requirements. This process typically necessitates that the robot be equipped with the ability for embodied exploration and manipulation. To achieve this goal, in this paper, we propose to incorporate multi-source knowledge to enhance the ability of robotic embodied exploration and manipulation. Specifically, to eliminate the inherent biases in decision-making of large language models (LLMs), we introduce a multi-source knowledge fusion module to generate more reasonable exploration sequences. Notably, grasping detection plays a critical role in the process of robot manipulation. To achieve a better balance between the accuracy and efficiency of the grasping detection network, we design a two-branch feature fusion module with residual blocks to improve network performance. Conditioned on the aforementioned innovations, the robot is capable of actively exploring and manipulating in constrained environments to meet human requirements. Extensive experiments are conducted in both simulation and real-world environments. The results demonstrate the effectiveness and efficiency of our proposed framework.