Speaker Adaptation for Lip Reading with Robust Entropy Minimization and Adaptive Pseudo Labels

Lip reading aims at recognizing what a speaker is saying in a silent video only based on visual information. With the development of deep learning, lip reading has attracted much attention. However, different people have different facial features, which leads to failure when the model is tested on unseen speakers. Most existing methods deal with this problem with fine-tuning or speaker adaptation methods which require the source domain data or the labels of the target domain. However, in real-world applications, due to privacy or storage issues and the difficulty in annotating the data, using these methods is not realistic. To cope with this problem, in this paper, we focus on test-time speaker adaptation for lip reading where the source data and the labels of target data are unavailable during adaptation. We propose a robust entropy minimization strategy to amplify the optimization of high-confident samples and reduce the influence of the low-confident samples. Furthermore, we propose an adaptive pseudo-label strategy to select more confident samples between the outputs from the source model and the augmented-averaged teacher model. Extensive experiments show that our method outperforms all the other baselines consistently, demonstrating its effectiveness and universality.