Multitask GANs for Semantic Segmentation and Depth Completion With Cycle Consistency

Semantic segmentation and depth completion are two challenging tasks in scene understanding, and they are widely used in robotics and autonomous driving. Although several studies have been proposed to jointly train these two tasks using some small modifications, such as changing the last layer, the result of one task is not utilized to improve the performance of the other one despite that there are some similarities between these two tasks. In this article, we propose multitask generative adversarial networks (Multitask GANs), which are not only competent in semantic segmentation and depth completion but also improve the accuracy of depth completion through generated semantic images. In addition, we improve the details of generated semantic images based on CycleGAN by introducing multiscale spatial pooling blocks and the structural similarity reconstruction loss. Furthermore, considering the inner consistency between semantic and geometric structures, we develop a semantic-guided smoothness loss to improve depth completion results. Extensive experiments on the Cityscapes data set and the KITTI depth completion benchmark show that the Multitask GANs are capable of achieving competitive performance for both semantic segmentation and depth completion tasks.