Depth-GASLT: Gloss-free Sign Language Translation Using Depth Information

Existing sign language translation research has improved performance by utilizing glosses in sign language datasets. However, due to the limitations of manually annotating glosses, gloss-free models have been proposed. Since these approaches do not teach glosses in a supervised manner, the model must infer glosses indirectly by utilizing spatially detailed information. However, previous studies have been limited by their inability to provide three-dimensional information of real-world space. To tackle this issue, we propose Depth-GASLT, a new architecture that estimates depth information from 2D sign language images and incorporates it to reflect three-dimensional position of the hands. Our method leverages depth information to enhance the translation process by accurately capturing the spatial nuances of hand movements in sign language. To demonstrate the effectiveness of our method, we conduct experiments on the PHEONIX-14 dataset and show that our method outperform the baseline model by 0.45 points based on the BLEU-4 score. In addition, the combining experiment showed that the element-wise combining method was effective. The results emphasize the significance of three-dimensional information in sign language translation, and are expected to contribute as a basis for effectively reflecting it.