3D ResESAM: A Sequential Attention Mechanism Enhanced 3D ResNet50 for Dog Emotion Recognition

An animal’s emotion is regarded as a critical indicator for assessing its welfare. At present, most canine emotion recognition models rely on static images, which leads to insufficient modeling of spatiotemporal features and low accuracy. To address this limitation, this study proposed a 3D ResESAM model based on 3D ResNet50, which is a pre-trained model on the Kinetics dataset. Specifically, the Enhanced Sequential Attention Module (ESAM) is integrated into the backbone to improve the capability of extracting and enhancing spatiotemporal features, and YOLO (You Only Look Once) was employed to detect and crop canine facial regions to reduce interference from complex backgrounds (e.g., furniture, humans, or other animals). To evaluate the effectiveness of 3D ResESAM, a canine emotion dataset named Dog-Face was constructed, which contains four emotions: happiness, sadness, anger, and calmness. Experimental results demonstrate that the proposed 3D ResESAM achieves an accuracy of 0.9239, representing a 3.25% improvement over the baseline 3D ResNet50. Furthermore, compared with other models such as C3D and R(2 + 1)D, whose accuracies are just 0.5215 and 0.8570, respectively, from different attention mechanism perspectives, the proposed model outperforms not only the baseline model without ESAM but also models incorporating other attention modules such as ECA and CBAM, with average accuracies being 0.8972 and 0.9103, respectively. In addition, cross-domain experiments are conducted on a human emotion dataset. The results show that 3D ResESAM achieves an average accuracy improvement of 2.39% over 3D ResNet50, demonstrating its significantly superior performance, effectiveness and potential in real-world applications.