Generating human facial animation by aggregation deep network and low-rank active learning with table tennis applications

The technique of generating facial animations based on emotions and speech significantly enhances various AI systems. This process begins with analyzing a speech signal to identify phoneme-emotion combinations, which are then translated into viseme-expression pairs for video animation. This study introduces a novel method for creating lifelike facial animations using emotional speech cues. We start by pinpointing specific acoustic features that accurately represent each phoneme-emotion pair. An active learning method is then applied to select key facial frames that effectively represent these pairs. During this selection phase, a deep learning model is designed to identify the most meaningful patches within each frame. Subsequently, these key frames are combined using the morphing technique, resulting in a fluid and visually appealing animation of facial expressions. The experiments demonstrate that this approach is capable of real-time performance on widely used mobile operating systems like iOS and Android, delivering animations that closely match the speech and emotional expressions. We further present the application of our technique to table tennis live streaming.