Multi-View 3D Reconstruction by Learning Robust Correspondence Structure With Shape Prior

Most multi-view 3D reconstruction pipelines reconstruct the 3D shape of a given object by aggregating features from multiple images. We believe that learning the correspondence structure across views conditioned on mining information from self-occluded parts within each view can enhance the effectiveness of feature aggregation. Unfortunately, this insight is yet to be investigated in current methods. This paper presents an approach termed Multi-view 3D Reconstruction by learning robust Correspondence structure with Shape Prior (M3RCS). It reconstructs the high-quality voxel representation from multiple RGB images. Firstly, a shape prior extraction network is devised in this paper. It applies a pre-trained prototype attentive model to capture global semantics for each view. Then, the shape prior of the given object is constructed via combining the global semantics from all views, promoting the mining of information from self-occluded parts. Secondly, a correspondence structure learning network is proposed to model an elementary correspondence structure among different views. Further, the network injects the constructed shape prior into the elementary correspondences, so as to learn a robust correspondence structure across all views despite the presence of self-occlusion in each view. This can realize the feature aggregation from multiple images effectively. Lastly, a fine-grained reconstruction network is designed, which progressively up-samples the aggregated feature and preserves useful information from the up-sampled result to produce a fine-grained 3D shape in voxel representation. Quantitative and qualitative experiments on the ShapeNet and Pix3D datasets verify that our M3RCS outperforms existing state-of-the-art methods.