What does this research mean for the field?

A deep learning-based sign language translation system integrating MediaPipe, Feedforward Neural Networks, and BiLSTM with an Attention Mechanism achieves 99% accuracy for ASL alphabet recognition and 78% for vocabulary recognition while operating in real-time on mid-range devices. Novelty: ClaimNovelty.METHODOLOGICAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To develop a real-time system for identifying and interpreting American Sign Language using deep learning techniques.

June 1, 2026Journal of Innovation and Community Engagement0 citationsOpen Access

Implementation of Hand Movement Tracking for Real-Time Sign Language Translation

BHBernardus Anggodho Aryudhawan HadiAtma Jaya University Yogyakarta WHWilfridus Bambang Triadi HandayaAtma Jaya University Yogyakarta SSS SuyotoAtma Jaya University Yogyakarta

Key Points

To develop a real-time system for identifying and interpreting American Sign Language using deep learning techniques.
Implemented a hand movement tracking system based on the ASL alphabet dataset and WLASL dataset
Utilized MediaPipe for extracting hand and body markers
Evaluated using accuracy, precision, recall, F1 Score, and K-fold cross-validation.
Achieved 99% accuracy for alphabet recognition
Achieved 78% accuracy for vocabulary recognition, significantly aided by the Attention Mechanism
System operates efficiently in real time at 15-20 FPS on mid-range devices.

Abstract

This study developed a real-time American Sign Language (ASL) sign language identification and interpretation system based on deep learning. The system used two data sources: the ASL alphabet dataset for individual character recognition and the WLASL dataset for vocabulary recognition. The WLASL dataset was chosen as the benchmark for evaluating complex word gestures because it encompasses a wide range of users and extensive movement dynamics. Data processing involved extracting hand-gesture and body-posture markers using MediaPipe, followed by preprocessing and augmentation. Two learning architectures were implemented: a Feedforward Neural Network for alphabet classification and a BiLSTM integrated with an Attention Mechanism for vocabulary recognition. The system was evaluated using accuracy, precision, recall, F1 Score, and K-fold cross-validation. The results demonstrated promising performance: 99% accuracy for alphabet recognition and 78% for vocabulary recognition, with the Attention Mechanism contributing substantially to vocabulary recognition. The system operates in real time at 15-20 FPS and is efficient on mid-range devices, potentially becoming an inclusive communication alternative for the sign language community.

Perguntar à IA

Bookmark

View Full Paper