3D-GS Talker: 3D Gaussian Based Audio-Driven Real-Time Talking Head Generation

Abstract In recent years, significant advancements have been made in the field of 3D talking head generation, yet achieving real-time rendering remains a persistent challenge critical for interactive applications. To address this, we present the 3D-GS Talker framework, a groundbreaking solution that leverages 3D Gaussian Splatting (3D-GS) to revolutionize real-time talking head synthesis. Unlike conventional approaches, which often compromise either speed or quality, our framework adopts an integrated methodology, driven by audio cues and guided by a Dynamic-Semantic-Gaussian (DSG) model, facilitating both flexibility and fidelity in video synthesis. Central to our framework is a Semantic Attention-based action decoder meticulously designed to ensure precise control over head movements, enabling natural and expressive animations. This is complemented by an innovative intermediate variable-based pre-alignment technique, synchronizing input audio with head motion to seamlessly integrate speech cues with facial expressions. Furthermore, to expedite the rendering process while maintaining visual integrity, we employ a rapid rasterization mechanism. Through comprehensive experimental evaluations, our framework demonstrates remarkable performance, achieving rendering speeds exceeding 100 frames per second (FPS) on average, meeting the stringent demands of real-time applications. Comparative analyses against state-of-the-art methods unequivocally highlight the superior rendering efficiency of 3D-GS Talker, showcasing a notable improvement of at least fivefold while preserving exceptional rendering quality across various metrics. In summary, the 3D-GS Talker framework represents a significant advancement in audio-driven talking head generation, offering unprecedented efficiency without compromising fidelity. By overcoming the longstanding challenge of real-time rendering, our framework opens new horizons for interactive applications spanning virtual assistants, gaming, telepresence, and beyond, where seamless integration of naturalistic human-machine interactions is paramount.