TWNet: a transfer learning-based framework for facial macro- and micro-expression spotting in long videos using hybrid features

In the field of computer vision, spotting the occurrences of macro-expression (MaE) and micro-expression (ME) intervals from long video sequences is an intriguing topic across various research domains. However, due to the small sample size, only the unsupervised learning methods had been applied to tackle this problem. In this article, transfer learning is introduced, proposing a novel Two-Way neural Network (TWNet) to handle the spotting tasks. TWNet learns hybrid features separately in each way: the optical flow features and the facial features in the apex frame. Each way is trained by the widely used MaE datasets first, then the video sequences are fed to TWNet. TWNet predicts the probability of each frame belonging to the MaE and forms a probability sequence. Then, a self-adaptive peak detection and interval generation algorithm is designed to select the apex frames in the probability sequence. Finally, the frames before and after the detected apex are labeled to generate the final MaE and ME intervals. Experiment results show that our TWNet outperforms the state-of-the-art models for spotting MaE and ME intervals on both the SAMM Long Videos and CAS(ME) 2 datasets.