Semantic Boundary Detection With Reinforcement Learning for Continuous Sign Language Recognition

Sign language recognition (SLR) is a significant and promising technique to facilitate the communication for the hearing-impaired people. In this paper, we are dedicated to weakly supervised continuous SLR, where for each sign video, there are only ordered gloss labels without temporal boundary along frames. To explicitly align video frames to the sign words in a sign video, we propose a novel semantic boundary detection method based on reinforcement learning for accurate continuous SLR. In our approach, we first propose a multi-scale perception scheme to learn discriminative representation for video clips. Then, we formulate the semantic boundary detection as a reinforcement learning problem. We define the state as the feature representation of a video segment, and the action as the determination of the semantic boundary's location. The reward is computed by the quantitative performance metric between the prediction sentence and the ground truth sentence. The policy network is trained with a policy gradient algorithm. Extensive experiments are conducted on CSL Split II and RWTH-PHOENIX-Weather 2014 datasets, and the results demonstrate the effectiveness and superiority of our method.