A silent speech reconstruction method based on Res-Conformer model

In response to the issues of low recognition accuracy in silent electromyographic facial action speech reconstruction tasks, this paper combines residual network (ResNet) and Transformer model to design a Res Transformer model based on ResNet structure and Transformer network. The model consists of three connected ResNet structures and several Transformer modules, and features are extracted using ResNet, The Transformer structure converts the electromyographic signal into a Mel frequency spectrum of 80 frequency bands, and sends the Mel frequency spectrum into the HiFiGAN network for speech reconstruction, ultimately obtaining the audible speech signal under silent action. In addition, our work also integrates acoustic speech signals, extracting the Mel frequency spectrum of synchronous acoustic speech signals as features, fully utilizing the time-frequency domain features of electromyography and speech signals. The experimental results show that the proposed Res Conform model has a phoneme recognition rate of 91.86% and a word error rate of 25.6%. Compared with using structures such as Transformer and LSTM, the Res-Conformer model has achieved effective improvement in recognition accuracy.