Due to substantial differences in appearance and movement between cartoon faces and real human faces, directly applying existing speech-driven speaker generation techniques to cartoon face images often fails to accurately capture key points, resulting in distorted expressions and animation artifacts. To address these challenges, this paper proposes a speech-driven cartoon face animation generation model called CartoonTalk, which integrates 2D cartoon key points with 3D motion coefficients. The model comprises three modules: face annotation, motion extraction, and 3D facial rendering. The face annotation module first annotates a cartoon image, capturing its 2D key points. Subsequently, the motion extraction module derives 3D motion coefficients for head and facial expressions from speech data. Finally, the 3D facial rendering module uses these 2D and 3D features to generate the animated cartoon face. Quantitative and qualitative evaluations demonstrate that our model significantly outperforms existing approaches in both lip-sync accuracy and visual quality, establishing new state-of-the-art benchmarks on cartoon datasets.
Wang et al. (Thu,) studied this question.