SignEdgeLVM transformer model for enhanced sign language translation on edge devices

Transformer architectures have accelerated the research in Continuous Sign Language Recognition and Translation (CSLRT), which involves predicting sign gloss patterns from video and converting them into spoken language. This process is challenging due to the lack of direct alignment between sign glosses and spoken words. While Transformers are effective due to their ability to process inputs in parallel, their high memory consumption makes the transformers less suitable for edge devices. To address this issue, we propose the SignEdgeLVM, a model that uses a Global Relative Attention Matrix (GRAM) and Dynamic Point Frame Sampling (DPFS) module. The SignEdgeLVM significantly lowers attention mechanism memory consumption by 78.22 MB (99.93 percent) per head in the attention layer for our implementation. Evaluated on the PHOENIX14T dataset, this optimization makes SignEdgeLVM suitable for edge devices.