Towards EEG-based Talking-face Generation for Brain Signal-driven Dynamic Communication

Research on decoding speech or generating images from human brain activity holds intriguing potential as neuroprosthesis for patients and innovative communication tools for general users. However, previous studies have been constrained in generating fragmented or abstract outputs, rendering them less applicable for serving as an alternative form of communication. In this paper, we propose an integrated framework that synthesizes speech from non-invasive speech-related brain signals and generates a talking-face that performs "lip-sync" using intermediate input decoded from brain signals. For realistic and dynamic brain signal-mediated communication, we generated a personalized talking-face by utilizing various forms of target data such as a real face or an avatar. Additionally, we performed a denoising process to enhance the quality of synthesized voices from brain signals, and to minimize unnecessary facial movements according to the noise. Therefore, clear and natural talking-faces, applicable to both real faces and avatars, could be generated from noisy brain signals, enabling dynamic communication. These findings serve as a pivotal contribution to the advancement of brain signal-driven face-to-face communication through the provision of integrated speech and visual interfaces. This represents a significant step towards the development of a more intuitive and dynamic brain-computer interface communication system.