Achieving State-of-the-art Accuracy in Arabic Sign Language Recognition with VideoMAE and Data Augmentation

The Arabic Sign Language (ArSL) plays a vital role in facilitating communication for deaf and hard-of-hearing individuals within Arabic-speaking communities, serving as a primary medium for social interaction, education, and access to services. However, the automatic recognition of ArSL remains a significant challenge due to its linguistic complexity, lack of standardized datasets, and the substantial variation in signing styles across regions, age groups, and social contexts. To address these challenges, we propose a word-level ArSL recognition system based on the transformer-based VideoMAE (masked autoencoder) model, which is pre-trained on the Kinetics-400 dataset and fine-tuned on a curated ArSL video dataset. We employ data augmentation techniques, such as downsampling, color inversion, and multiplication, which simulate real-world variability in gesture appearance and recording conditions, to enhance the model’s adaptability to real-world signing variations. Our system achieved a 97% accuracy, outperforming the 49% baseline and surpassing the traditional convolutional neural network-, recurrent neural network-, and long short-term memory-based approaches. Additionally, the system demonstrated consistent performance across 15 data subsets and supported real-time inference at 30 frames per second on GPU hardware. This performance demonstrates the effectiveness of VideoMAE’s transformer-based architecture and data augmentation in capturing the spatial–temporal patterns of sign language. It offers a robust foundation for future research in sentence-level and real-time sign language translation. Lastly, the results contribute to the development of inclusive communication technologies for Arabic-speaking deaf and hard-of-hearing communities.