Visual Speech Recognition Using 3D CNN and LSTMS

Lip reading, a challenging task within the domain of multimodal human-computer interaction, has garnered significant interest due to its potential applications in various fields, including assistive technologies, security systems, and human-computer interfaces. This research paper presents a deep learning-based approach to lip reading, leveraging Convolutional Neural Networks (CNNs) for feature extraction from lip images and employing a Bidirectional Long Short-Term Memory (Bi-LSTM) network for sequence modelling and transcription. The model integrates a 3D CNN architecture for spatiotemporal feature extraction, allowing it to capture spatial and temporal dependencies in lip movements. Training is facilitated using the Connectionist Temporal Classification (CTC) loss function, enabling end-to-end learning. Experimental results on the Grid Corpus dataset demonstrate the effectiveness of the proposed approach, achieving an impressive transcription accuracy rate of 85.65%. This high accuracy showcases the model's ability to accurately interpret subtle visual cues of lip movements and transcribe speech with high precision. Furthermore, evaluation metrics such as Word Error Rate (WER), Character Error Rate (CER), and overall accuracy provide insights into the model's performance. This research contributes to advancing automatic lip-reading systems, offering a deeper understanding of the challenges and opportunities in this domain. The proposed approach, combining 3D CNNs and recurrent networks, lays the foundation for future developments in improving the accuracy and robustness of lip-reading systems, ultimately enhancing their usability and applicability in diverse real-world scenarios.