3-D Semantic Terrain Reconstruction of Monocular Close-Up Images of Martian Terrains

Martian surface, as a typical unstructured terrain, is extremely challenging for Mars exploration missions. Commonly, Mars rovers require multiple sensors to explore such harsh environment, such as depth cameras, range finder and other devices. However, the onboard load, power and storage of rovers are not sufficient to achieve high-level stereoscopic perception, which can be adverse to downstream tasks such as visual navigation and scientific exploration. To this end, in this paper we propose a high-level awareness perception light-weight framework using only close-shot monocular images to implement semantic 3D reconstruction of Martian landforms. This framework consists of two parts. One is a semantic segmentation module based on the proposed real-time Mars terrain segmentation (RMTS) network to extract intra-class and inter-class contexts by local supervision. The other is a depth generation module based on a dual-encoder pix2pix network to encode the visual and semantic information of monocular images, simultaneously. To validate the proposed framework, we construct a Martian planar-stereo dataset based on AI4Mars, an open-source semantic segmentation dataset for Mars surface. It contains monocular close-up Martian images, semantic images and depth images that match each other. After training, the accuracy of proposed semantic segmentation model can reach 84.0% mIoU, with 152.2 FPS on a single RTX6000-24GB GPU. The absolute relative error of pixels in depth images between generation model and the ground truth is 0.367, while the root mean square error gets to 0.510, and the accuracy is 0.753 with 42.9 FPS. The overall environment perception scheme is with 9.5FPS.