Dynamic Audio-Visual Speech Enhancement using Recurrent Variational Autoencoders

Audio-visual speech enhancement (AVSE) has emerged as a promising approach to enhance speech perception in noisy environments by leveraging both auditory and visual cues present in speech signals. We propose an audio-visual recurrent variational autoencoder (AV-RVAE) for speech enhancement. This model incorporates visual lip motion along with audio to improve speech quality. We extend the RVAE model, which has shown superior performance in audio-only speech enhancement, to handle audio-visual data. Our approach uses both a dedicated video encoder for visual feature extraction and a recurrent neural network (RNN) for temporal modeling. Additionally, we introduce both causal and non-causal variants of AV-RVAE to capture temporal dynamics of speech data. Experimental results on the NTCD-TIMIT dataset show that AV-RVAE outperforms existing methods, showing improvements in scale-invariant signal-to-distortion ratio (SISDR), perceptual evaluation of speech quality (PESQ), and short-term objective intelligibility (STOI) metrics.