Video Emotion Analysis: Segmentation, Classification, and Explainability Techniques in Review

Comprehensive review of multimodal emotion recognition techniques and trends Comparative analysis of ML and deep learning models for emotion classification Identifies key feature extraction and fusion strategies across modalities Highlights challenges, limitations, and open research gaps in real-world systems Summarizes future directions for robust and scalable emotion recognition models Video emotion analysis is a complex and rapidly evolving research domain challenged by temporal dynamics, multimodal heterogeneity, dataset bias, and the persistent affective gap between computational features and human emotional perception. This survey systematically reviews 186 studies addressing segmentation, classification, multimodal fusion, and explainable artificial intelligence (XAI) in video-based emotion recognition. The analysis indicates that deep learning architectures, particularly spatio-temporal and attention-based models, demonstrate clear performance advantages over traditional machine learning approaches; however, their generalization capability remains strongly influenced by dataset design and evaluation protocols. Multimodal fusion is shown to enhance robustness in real-world and noisy conditions, though its effectiveness depends critically on alignment strategies and modality quality. Furthermore, explainability techniques are predominantly implemented as post hoc mechanisms rather than being intrinsically integrated into model design, thereby limiting their contribution to trustworthy deployment in sensitive applications. To address these limitations, a unified end-to-end pipeline framework is proposed to systematically connect segmentation, fusion, prediction, and explanation stages, providing a structured roadmap for advancing robust, scalable, and transparent video emotion analysis systems.