Towards realism, controllability, and affordability in learning-based 3D head creation

Digital 3D head models are widely used in many applications, including video games, virtual reality, telepresence, and film production. There are three main objectives (also challenges) in 3D head creation: controllability, realism, and affordability. Although existing methods can achieve individual objectives, it remains challenging to satisfy multiple objectives together. For example, the 3D head models used in films can be photo-realistic, but they are very costly. In this thesis, we study how to achieve these objectives in 3D head creation through learning-based methods. We start with the controllability objective and address 3D face shape creation that allows face-editing and stylization. We first propose the NEO-3DF approach, which uses sub-models to represent semantic facial parts (e.g., the nose). This design enables intuitive editing and improves alignment of the created 3D face with the 2D input image. Then, we introduce a weakly supervised 3D face stylization method that requires only one template face (artist-designed style) and transfers facial features from a reconstructed 3D face to this template. Besides controllability, these methods also enable affordability in 3D face reconstruction, editing, and stylization. Next, we study how to achieve the realism objective and address 3D-consistent face image synthesis. To this end, we propose the StyleMorpheus method, which is trained with the objective of generating photorealistic face images. StyleMorpheus enables independent control over facial identity, expression, and appearance through a disentangled style-code space. The training of StyleMorpheus requires only in-the-wild 2D face images, which enables affordability by lowering the data collection costs. Finally, we extend our study to jointly achieve controllability, realism, and affordability by addressing photorealistic and animatable 3D head avatar creation. We propose two 3D Gaussian Splatting (3DGS)–based methods: Déjà-vu and ArchitectHead. Déjà-vu first learns a generalized reconstruction model from 2D face images, which is then used to provide a strong initialization for efficient personalization using a short monocular video of a given individual. ArchitectHead further addresses the run-time trade-off between rendering quality and efficiency in 3DGS-based head avatars. Together, these methods enable the creation of highly realistic, controllable, and cost-effective 3D head avatars suitable for real-time applications.