Answers are grounded in this paper and cite their sources.
This paper proposes an Audio-Visual Speech Recognition (AVSR) model using both audio and visual speech informationto improve recognition accuracy in a clean and noisy environment. Mel frequency cepstral coefficient (MFCC) and DiscreteCosine Transform (DCT) are used to extract the effective features from audio and visual speech signal respectively. TheClassification process is performed on the combined feature vector by using one of main Deep Neural Network (DNN)architecture, Bidirectional Long-Short Term Memory (BiLSTM), in contrast to the traditional Hidden Markov Models (HMMs).The effectiveness of the proposed model is demonstrated on a multi-speakers AVSR benchmark dataset named GRID. Theexperimental results show that the early integration between audio and visual features achieved an obvious enhancement in therecognition accuracy and prove that BiLSTM is the most effective classification technique when compared to HMM. The obtainedresults when using integrated audio-visual features achieved highest recognition accuracy of 99.07%, this result demonstrates anenhancement of up to 9.28% over audio-only recognition for clean data. While for noisy data, the highest recognition accuracy forintegrated audio-visual features is 98.47% with enhancement up to 12.05% over audio-only. The main reason for BiLSTMeffectiveness is it takes into account the sequential characteristics of the speech signal. The obtained results show the performanceenhancement compared to previously obtained highest audio visual recognition accuracies on GRID, and prove the robustness ofour AVSR model (BiLSTM-AVSR).
No takes yet. Share an insight, caveat, or question.
Elmaghraby et al. (2019) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: