This is the authors' abstract. We don't add key points for this paper.
The proposal of audio-visual speech recognition (AVSR) combining acoustic and visual features has caused heated debates with the growth of online conferences. Unfortunately, due to the fact that audio and visual information usually affect each other, the recognition rate cannot improve. In this paper, we describe an improved audio-visual speech recognition network called AVSR-SA-MASK which does well in fusing. Therefore, this method is robust to low SNR scenes and face occlusion scenes. The mask fusion module of AVSR-SA-MASK can capture the correlations along primitive and extracted dimensions for both the audio and the visual. Instead of utilizing single loss at the training stage, we also introduce a balanced loss to improve modal identification abilities. In TCD-TIMIT and LRS2, the experiment results show that char error rate (CER) is improved by 15% to 31% relatively.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Li et al. (2022) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: