Lip-synchronized Talking Face Generation with Enhanced Mouth Movement

Over the last few years, there have been significant advancements in the technology of generating facial videos driven by audio. Especially in the case of individual-specific general methods, there have been significant breakthroughs in generating facial videos that preserve personal identity while realistically mimicking speech. The current general methods for individuals still fall short in representing lip movements, often resulting in the phenomenon of insufficiently pronounced lip movements during speech. To address this issue and enhance the realism of generated facial videos, we have introduced a mouth enhancement module on top of existing approaches. This module aims to improve the shortcomings of individual-specific general methods in representing mouth movements. In the first stage, we derive a facial speaking sketch based on input audio using a transformer. In the second stage of video rendering, we integrate a mouth enhancement module to reinforce lip movements during training, making the final generated facial speaking videos more lifelike. During this process, the input audio is repeatedly utilized to ensure synchronization between the generated video and audio.