A Closed-Loop Framework for Grasping Unknown Objects by Coupling Vision-Based DRL, Real-to-Sim Adaptation, and IBVS

We propose a novel zero-shot learning framework for closed-loop 4-DoF grasping of unknown objects in an uncluttered environment by coupling data-driven learning models with classical robot control. It combines model-free, raw vision-based deep reinforcement learning (DRL), trained purely in simulation, with real-to-sim domain adaptation (DA) via generative adversarial networks (GANs), trained on unpaired datasets, and with classical image-based visual servoing (IBVS). We introduce a DRL grasping agent fed with RGB-D images and trained in a fully interactive simulation environment, providing continuous velocity control actions. We trained the proximal policy optimization (PPO) agent to grasp 70 objects in 512 parallel simulation environments using four-channel simulated camera-rendered images. For robustness to visual variations, we applied domain randomization to illumination, textures, and colors of the environments and objects to be grasped. To transfer the control policy to the real world, we collected 5500 unpaired images from each domain and trained a CycleGAN-based DA agent. We then fed the real-to-sim-adapted images to the control agent. The high variation in the backgrounds used in the simulation and real-world datasets allowed our framework to perform successfully in the presence of textured backgrounds. The IBVS component further increases grasping performance through precise final grasp-pose refinement. Our framework does not require any real-world grasp examples during training, nor any fine-tuning in the real world. It achieves a mean positioning success (PS) rate of 98.88% and a mean zero-shot grasp success rate of 95.55% on previously unseen objects using a standard parallel jaw gripper.