Multimodal Speaker Recognition using voice and lip movement with decision and feature level fusion

The speech generation mechanism is fundamentally bimodal in nature. It is an audio and visual representation. Including visual information obtained from the lip movement of a speaker, in addition to the voice is hence justified for a text independent automatic speaker recognition system (ASR). Additionally, lip movement information is invariant to acoustic noise perturbation making the system more robust. Hence, we were motivated to design a dynamic audio-visual speaker recognition system. The objective of this research is to identify a speaker from its voice regardless of the spoken content and strengthen the accuracy of recognition. Classical methods and state of art neural networks has been employed to accomplish this. The learning model for voice modality was computed by concatenating dynamic features to the handcrafted features, which were further optimized using Fisher score technique, leading to improvement in speaker recognition. Support Vector Machines (SVM) and Convolution Neural Network (CNN) classifiers gave a competitive accuracy of 94.77%. For extracting information from the lip movement, Histogram of Gradient (HOG) feature detector algorithm was implemented on the image frames obtained from the video. Unique lip movements were was also computed from the mouth region landmark points of Facial Landmarks. Multimodal framework was accomplished by feature level fusion of voice and lip features with CNN as classifier. The significance of the proposed work lies in the novel use of CNN for speech features. The authors have successfully demonstrated that lip movement features help in liveness detection along with automatic speaker recognition (ASR). The proposed method achieves 91.4% testing accuracy in comparison to the state-of-the-art method.