Learning 3D Face Reconstruction From the Cycle-Consistency of Dynamic Faces

Reconstructinga 3D face from a single image is a crucial task in numerous multimedia applications. Face images with ground-truth 3D face shapes are scarce, so unsupervised deep learning methods, which rely primarily on the free supervision signal derived from the visual disparity between the input image and the rendered counterpart of the predicted 3D face, have proven superior for reconstructing 3D faces. However, it is challenging for such techniques to decouple the dynamic 3D face properties such as pose or expression from a single 2D image, especially when similar local visual appearance changes can be caused by both pose and expression motion, resulting in imprecise 3D face reconstruction. In this article, a novel cycle-consistency in dynamic 3D face characteristics is introduced as a free supervisory signal for learning accurate 3D face shapes from unlabeled facial images. The main idea of cycle-consistency is to explicitly inject the head pose or facial expression variation between video frames into a face image, and then to extract and reverse the injected variation in order to reconstruct the face image to its original state. In our model, a CNN network with multiple branches is proposed to disentangle 3D face properties like identity, expression, pose, and texture from 2D facial images, one branch for each 3D face property. During training, our model learns to completely decouple the dynamic 3D face properties (pose and expression) to be useful for performing cycle-consistent face reconstruction. Extensive experiments demonstrate the superiority of our approach. On the challenging AFLW2000-3D, MICC Florence, and NoW datasets, our method outperforms or is on par with the state of the art.