Multimodal 3D measurement setup for generating multimodal real-world data sets for AI-based transparent object recognition

Multimodal 3D imaging is a key technology in various areas, such as medical technology, trust-based human-robot collaboration and material recognition for recycling. This technology offers new possibilities, particularly for the 3D perception of optically uncooperative surfaces in the VIS and NIR spectral range, e.g., transparent or specular materials. For this purpose, a thermal 3D sensor developed by Landmann et al. allows the 3D detection of transparent and reflective surfaces without object preparation, which can be used to generate real multimodal 3D data sets for AI-based methods. The 3D perception of optically uncooperative surfaces in VIS or NIR is still nowadays an open challenge (cf. Jiang et al.). However, to overcome this challenge, we have developed a new measurement principle TranSpec3D, with which we can generate real multimodal 3D data sets with annotation without object preparation techniques. This system significantly reduces the effort required for data acquisition. We also show the advantages and disadvantages of our extended measurement principle and data set compared to other data sets (generated with object preparation).