What question did this study set out to answer?

This research aims to enhance translation of spoken Arabic text into Arabic Sign Language (ArSL) using a transformer model.

May 25, 2026Open Access

Transformer Model for Direct Translation of Spoken Text to Arabic Sign Language (ArSL)

Key Points

This research aims to enhance translation of spoken Arabic text into Arabic Sign Language (ArSL) using a transformer model.
Developed a neural machine translation approach with a transformer-based model for ArSL.
Utilized an encoder-decoder architecture for text representation and gesture generation.
Evaluated the model with quantitative metrics like mean square error and dynamic time warping, along with qualitative assessments from Deaf community members.
The enhanced model reduced dynamic time warping (DTW) by 43.4% for non-continuous ArSL tests.
Achieved a 2.25% reduction in DTW for continuous ArSL tests using Short Sermon data.
Received 75.2% user satisfaction rating, indicating high acceptance among users.

Abstract

Arabic Sign Language (ArSL) is a formal language used by Deaf and Hard of Hearing communities. It helps them understand Arabic spoken by hearing people, particularly in formal speech, such as news or sermons. The production of sign language systems can support communication by translating spoken languages into ArSL videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures and limited ability to capture the textual meaning of speech. This research focused on a neural machine translation (NMT) approach to convert spoken Arabic texts into Arabic Sign Language (ArSL) using a transformer-based model to overcome these difficulties. The ArSL video of the sign is produced as a series of 2D skeletal poses. The proposed methodology is based on an encoder-decoder architecture: texts are represented by a transformer model, and a posture decoder generates gestures. The model is evaluated using quantitative criteria, such as mean square error (MSE) and the dynamic time warping (DTW) measure, as well as qualitative evaluations by sign language experts and users from the Deaf and Hard of Hearing community. The study uses Friday sermon data as a reference to build the translation model and applies optimization techniques such as temporal modulation, noise addition, and resizing to enhance performance. The results show that the enhanced model improves the quality of gesture representation, reducing DTW by 43.4% relative to the baseline in the non-continuous ArSL test. On the other hand, in the Continuous ArSL test using Short Sermon, the DTW is reduced by around 2.25%. In addition, the enhanced model received high user acceptance, with 75.2% satisfaction. Moreover, unified qualitative evaluation criteria for sign language production are proposed, including clarity, accuracy, consistency, and naturalness, which could serve as a basis for future evaluations in this field. These criteria are used to evaluate the generated ArSL and measure user satisfaction.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper