A Simulated Two-Stream Network via Multi-levelDistillation of Reviewed Features and DecoupledLogits for Video Action Recognition

Abstract Video-based action recognition, which models human activities and classifiesactions from video frame sequences, offers significant utility in computer visionand computer graphics applications. The methods based on two-stream architectureare commonly used, which consist of an appearance stream for RGBvideo frames and a motion stream for optical flow frames. Due to the highcomputational complexity of optical flow, recent studies have attempted to useoptical flow only in the training phase and avoid computing optical flow in thetesting phase. However, these approaches fail to fully leverage the benefits ofoptical flow during training, leading to limited improvements in accuracy. Toaddress this issue, we introduce a Simulated Two-Stream Network (STS-Net),utilizing a multi-level knowledge distillation approach to capture motion representations.First, we try to distill motion feature of optical flow across variouslevels through a review mechanism, thereby capturing both low-level feature andhigh-level semantic information. Second, we apply a decoupled logit distillationloss to obtain a more comprehensive knowledge transfer. Additionally, we analyzethe role of the activation function in fusing the two streams, and propose aneffective fusion strategy named “ActivNo”. The experimental results on benchmarkdatasets (i.e., HMDB51, UCF101, and Kinetics400) demonstrated that theexploited model STS-Net achieves superior performance, surpassing comparable methods in terms of both efficiency and accuracy. The code is openly availableat https://github.com/BiBiKo219/STS-Net.