Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects

The basis of many object manipulation algorithms is RGB-D input. Yet, RGB-D sensors can only provide distorted depth maps for a wide range transparent objects due light refraction and absorption. To tackle the challenges posed by transparent objects, we propose TranspareNet, a point cloud and depth completion method, with the ability to complete the of transparent objects in cluttered and complex scenes, even with filled fluid contents within the vessels. To address the shortcomings existing transparent object data collection schemes in literature, we also an automated dataset creation workflow that consists of-controlled image collection and vision-based automatic annotation. this automated workflow, we created Toronto Transparent Objects Depth (TODD), which consists of nearly 15000 RGB-D images. Our experimental demonstrates that TranspareNet outperforms existing state-of-the-art completion methods on multiple datasets, including ClearGrasp, and that also handles cluttered scenes when trained on TODD. Code and dataset will be at https://www.pair.toronto.