Database‐Assisted Object Retrieval for Real‐Time 3D Reconstruction

Abstract In recent years, real‐time 3D scanning technology has developed significantly and is now able to capture large environments with considerable accuracy. Unfortunately, the reconstructed geometry still suffers from incompleteness, due to occlusions and lack of view coverage, resulting in unsatisfactory reconstructions. In order to overcome these fundamental physical limitations, we present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real‐time. With this approach, we are able to replace scanned RGB‐D data with complete, hand‐modeled objects from shape databases. We align and scale retrieved models to the input data to obtain a high‐quality virtual representation of the real‐world environment that is quite faithful to the original geometry. In contrast to previous methods, we are able to retrieve objects in cluttered and noisy scenes even when the database contains only similar models, but no exact matches. In addition, we put a strong focus on object retrieval in an interactive scanning context — our algorithm runs directly on 3D scanning data structures, and is able to query databases of thousands of models in an online fashion during scanning.