FFDB: feature fusion with debiased 3D landmark encoding and micro-expression augmentation for multimodal emotion recognition in e-learning

Recognising student emotions in an online e-learning environment becomes essential for understanding the effects of a particular teaching process. Within educational systems, the ability to recognise emotions from video is crucial for enabling real-time instructional feedback, adaptive content delivery, demographic balances and personalised engagement strategies. This research introduces a multimodal spatiotemporal architecture that utilises facial geometry, texture, and motion through three parallel branches, extracting facial curvature, biomechanical features, and motion features. Every modality is independently processed through a multi-stage temporal encoder, comprising One-Dimensional Convolutional Neural Network (1D CNNs) for modelling short-range dependencies, Bidirectional Long Short-Term Memory (LSTM) for mid-range temporal learning, and Transformer blocks for long-range attention modelling. Feature fusion with a debias layer and optimisation is driven by a novel enhanced Dual-Wing Loss, integrating additive angular margin, Wing, and Focal loss (α = 0.25, γ = 2.0) to improve class separability and address imbalance. Stochastic Depth regularisation prevents overfitting, while micro-expression synthesis enriches minority-class data by interpolating between neutral and expression states. Trained on a balanced Dataset for Affective States in E-Environments (DAiSEE) subset (387 clips), the model achieves 99.36% validation accuracy, with macro precision, recall, and F1-scores 0.99, and Area Under the Curve (AUC) = 1.00 across all classes.