Temporal Video Segmentation Approach for Peruvian Sign Language

Continuous Sign Language Recognition is a task that involves the recognition of complete sequences of Sign Language videos and translating them into spoken or written language. This task is particularly challenging due to the complex temporal structure and variability in signing styles. In this context, temporal video segmentation emerges to distinguish individual signs from transitions in continuous video streams. Despite its importance, temporal segmentation for Sign Language has not yet fully benefited from recent advancements in Machine Learning. Many existing approaches still rely on outdated architecture. This study aims to learn the characteristics that distinguish signs from transitions, which are fundamental elements of Sign Language. To this end, we adapt two current temporal segmentation models, DiffAct and MS-TCN, and apply them to our own precisely annotated datasets for Peruvian Sign Language. We explore three training strategies—baseline, data augmentation, and multi-dataset. Results suggest that they can enhance the scores for both models but at the cost of increased variability across splits. Notably, the diffusion-based model showcased its ability to generalize to unseen sequences through higher scores for sign and transition identification on the test set, reaching a median value of 71.89% for mF1S and 72.68% for mF1B.