Automatic 3D building reconstruction from multi-view aerial images with deep learning

The study presented in this paper introduced a new fully automatic three-dimensional building reconstruction method that can generate first level of detail (LoD 1) building models from multi-view aerial images without any assistance from other data. The accuracy and completeness of our reconstructed models have approached that of manually delineated models to a large extent. The presented method consists of three parts: (1) efficient dense matching and Earth surface reconstruction, (2) reliable building footprint extraction and polygon regularization, and (3) highly accurate height inference of building roofs and bases. First, our novel deep learning-based multi-view matching method, composed of a convolutional neural network, gated recurrent convolutions, and a multi-scale pyramid matching structure, is used to reconstruct the digital surface model (DSM) and digital orthophoto map (DOM) efficiently without generating epipolarly rectified images. Second, our three-stage 2D building extraction method is introduced to deliver reliable and accurate building contours. Deep-learning based segmentation, assisted with DSM, is used to segment buildings from backgrounds; and the generated building maps are fused with a terrain classification algorithm to reach better segmentation results. A polygon regularization algorithm and a level set algorithm are thereafter employed to transfer the binary segmentation maps to structured vector-form building polygons. Third, a novel method is introduced to infer the height of building roofs and bases using adaptive local terrain filtering and neighborhood buffer analysis. We tested our method on a large experimental area that covered 2284 aerial images and 782 various types of buildings. Our results as far as correctness and completeness exceeded the results of other similar methods in a between-method comparison by at least 15% for individual 3D building models with many of them comparable to manual delineation results.