Temporal Keyframe Technique based on CNN and LSTM for Enhancing Lip Reading Performance

The purpose of this research is to present a method for improving lip reading performance through the analysis of temporal keyframes using CNN and LSTM. The performance of the model is determined by employing the input data from the data preparation process. The lip image data is divided into three groups for development: 3 frames, 5 frames, and 10 frames, which include half-lip image data. Lip reading research has never been conducted before based on the assumption of a typical human body shape with left and right symmetry, as in this study. When the unseen test set was evaluated, 10 frames achieved the best recognition rate, with results of 0.879% accuracy for the full lip image dataset and 0.868% accuracy for the half lip image dataset, exhibiting comparable performance.