AU‐Guided Feature Aggregation for Micro‐Expression Recognition

ABSTRACT Micro‐expressions (MEs) are spontaneous and transient facial movements that reflect real internal emotions and have been widely applied in various fields. Recent deep learning‐based methods have been rapidly developing in micro‐expression recognition (MER).Still, it is typical to focus on the one‐sided nature of MEs, covering only representational features or low‐ranking Action Unit (AU) features. The subtle changes in MEs characterize its feature representation weak and inconspicuous, making it tough to analyze MEs only from a single piece or a small amount of information to achieve a considerable recognition effect. In addition, the lower‐order information can only distinguish MEs from a single low‐dimensional perspective and neglects the potential of corresponding MEs and AU combinations to each other. To address these issues, we first explore how the higher‐order relations of different AU combinations correspond with MEs through statistical analysis. Afterward, based on this attribute, we propose an end‐to‐end multi‐stream model that integrates global feature learning and local muscle movement representation guided by AU semantic information. The comparative experiments were performed on benchmark datasets, with better performance than the state‐of‐art methods. Also, the ablation experiments demonstrate the necessity of our model to introduce the information of AU and its relationship to MER.