Continuous Sign Language Recognition using Convolutional Neural Network

Automatic Continuous Sign Language Recognition is a task that aims at associating words/phrases for sign language gestures. The scope of building a CSLR framework extends beyond bridging the communication barrier between the deaf community and the hearing majority but includes profound benefits such as bootstrapping translation systems, human-computer interaction, virtual reality environments, etc. In this study, a continuous sign language recognition framework based on a CNN architecture is proposed. Utilizing the Argentina Sign Language dataset, sign language gestures are preprocessed to extract training and testing frames. Employing transfer learning with the ResNet-50 model, a CNN is trained to classify each frame into one of 64 different words present in the dataset. The paper also conducts a thorough post-training data analysis, offering insights into areas where the model performed less effectively, identifying commonly confused pairs of sign language gestures. Additionally, a heatmap overlay technique is employed to elucidate the black-box CNN model.