Explicitly modeling trajectories and correlations for video analysis

Video analysis tasks such as action recognition and sign language recognition require that both the appearance and dynamics of objects and people of interest are captured. Extraction of distinctive local appearance changes of regions under motion, such as finger movement while gesturing, is often essential to make correct classifications. Without disentangling gross and fine spatio-temporal patterns, both will be confounded, leading to reduced accuracy. We introduce two innovations to tackle this issue. First, we explicitly model motion of image patches by temporally aligning visual tokens using optical flow. In addition to obtaining a coarse motion representation, for a query token, we apply self-attention along the trajectory. This essentially cancels out gross movement and makes possible the extraction of distinctive local patterns of regions under motion. Our second innovation is a dynamic attention mechanism that filters out irrelevant frame regions. It assigns dynamic key–value tokens from correlated regions to each query to focus on coordinated appearance changes such as joint hand and mouth movements while gesturing. We combine these innovations in a novel Trajectory and Correlation (TC) block, a hybrid network that effectively models spatio-temporal information from trajectories and correlated regions. We experiment on four sign language (PHOENIX14, PHOENIX14-T, CSL, and CSL-Daily) and two action recognition (Kinetics-400 and Something-Something V2) datasets. Using TC blocks in different backbones consistently achieves improved performance with a modest increase in parameters, and 30% additional computational cost.