High Resolution Acquisition, Learning and Transfer of Dynamic 3‐D Facial Expressions

Abstract Synthesis and re‐targeting of facial expressions is central to facial animation and often involves significant manual work in order to achieve realistic expressions, due to the difficulty of capturing high quality dynamic expression data. In this paper we address fundamental issues regarding the use of high quality dense 3‐D data samples undergoing motions at video speeds, e.g. human facial expressions. In order to utilize such data for motion analysis and re‐targeting, correspondences must be established between data in different frames of the same faces as well as between different faces. We present a data driven approach that consists of four parts: 1) High speed, high accuracy capture of moving faces without the use of markers, 2) Very precise tracking of facial motion using a multi‐resolution deformable mesh, 3) A unified low dimensional mapping of dynamic facial motion that can separate expression style, and 4) Synthesis of novel expressions as a combination of expression styles. The accuracy and resolution of our method allows us to capture and track subtle expression details. The low dimensional representation of motion data in a unified embedding for all the subjects in the database allows for learning the most discriminating characteristics of each individual's expressions as that person's “expression style”. Thus new expressions can be synthesized, either as dynamic morphing between individuals, or as expression transfer from a source face to a target face, as demonstrated in a series of experiments. Categories and Subject Descriptors (according to ACM CCS): I.3.7 Computer Graphics: Animation; I.3.5 Computer Graphics: Curve, surface, solid, and object representations; I.3.3 Computer Graphics: Digitizing and scanning; I.2.10 Artificial intelligence: Motion ; I.2.10 Artificial intelligence: Representations, data structures, and transforms; I.2.10 Artificial intelligence: Shape; I.2.6 Artificial intelligence: Concept learning