Real‐Time Recognition of Dynamic Gestures in Pakistan Sign Language Using YOLO and LSTM Networks

Sign language is a primary communication modality for many deaf and hard‐of‐hearing individuals, yet many recognition systems remain limited by static‐gesture assumptions and weak generalization to realistic acquisition variability. This paper presents a real‐time dynamic Pakistan Sign Language (PSL) gesture recognition framework that combines YOLOv8 for hand localization with recurrent temporal modeling for sequence classification. A dynamic PSL dataset was collected from 20 participants covering 20 gesture classes (2220 RGB and Kinect video sequences), recorded across five sessions, multiple environments, three lighting conditions (bright, moderate, and dim), and three capture distances (2, 3, and 4 ft). To address generalization concerns, we report signer‐independent leave‐one‐subject‐out (LOSO) evaluation with mean ± standard deviation, along with complementary cross‐session and cross‐environment tests, and an aggregate confusion matrix computed over the independent test predictions. Under LOSO, the proposed temporal model using bounding box and velocity features achieves 93.04% ± 1.61% accuracy with macro‐F1 of 0.930 ± 0.016. Cross‐session testing (train sessions 1–4 and test session 5) yields 95.62% ± 0.84% accuracy, while cross‐environment testing (leave‐one‐environment‐out) achieves 91.18% ± 1.94% accuracy, quantifying performance under domain shift. We further provide controlled baselines and ablations, including YOLO‐only majority voting, CNN + LSTM on YOLO‐cropped hand images, a lightweight 3D CNN on short clips, and a GRU variant, confirming the contribution of temporal modeling and motion cues. The system maintains accuracy above 90% under varying lighting and distance conditions, supporting practical real‐time deployment. Limitations include the current focus on manual hand gestures only; future work will integrate nonmanual cues (face and body pose) for more linguistically complete PSL processing.