Unsupervised deep depth completion with heterogeneous LiDAR and RGB-D camera depth information

• The first unsupervised LiDAR-depth fusion depth completion method, named LidarDepthNet. • Only depth input is required, making it suitable for application in darkness. • LidarDepthNet effectively fuses heterogeneous depth to obtain high-quality depth images. • LidarDepthNet outperformed comparative methods under real-world and synthetic datasets. • The first dataset for heterogeneous depth information fusion was provided. In this work, a depth-only completion method designed to enhance perception in light-deprived environments. We achieve this through LidarDepthNet, a novel end-to-end unsupervised learning framework that fuses heterogeneous depth information captured by two distinct depth sensors: LiDAR and RGB-D cameras. This represents the first unsupervised LiDAR-depth fusion framework for depth completion, demonstrating scalability to diverse real-world subterranean and enclosed environments. To facilitate unsupervised learning, we leverage relative rigid motion transfer (RRMT) to synthesize co-visible depth maps from temporally adjacent frames. This allows us to construct a temporal depth consistency loss, constraining the fused depth to adhere to realistic metric scale. Furthermore, we introduce measurement confidence into the heterogeneous depth fusion model, further refining the fused depth and promoting synergistic complementation between the two depth modalities. Extensive evaluation on both real-world and synthetic datasets, notably a newly proposed LiDAR-depth fusion dataset, LidarDepthSet, demonstrates the significant advantages of our method compared to existing state-of-the-art approaches.