Emotion Recognition from rPPG via Physiologically Inspired Temporal Encoding and Attention-Based Curriculum Learning

Figures

• 

  Figure 5. Qualitative analysis of sparse attention mechanism on ECG signal. This figure presents a comparative qualitative analysis of the attention distributions from a standard SoftMax model and our proposed α-Entmax sparse attention model on a representative ECG signal from a random subject’s random session. (a) displays the ECG signal with physiologically meaningful R-peaks highlighted. (b) shows the diffuse attention distribution of SoftMax (Entropy: 2.92). (c) illustrates the highly sparse attention distribution of α-Entmax (Entropy: 1.14). (d) overlays both attention distributions on the ECG signal, demonstrating how α-Entmax selectively focuses on specific physiological events, particularly R-peak clusters.

• 

  Figure 1. End-to-end pipeline for rPPG-based emotion recognition. An RGB video is separated into 128-frame segments (≈4 s), converted to rPPG waveforms using a pretrained PhysMamba, encoded using the Multi-scale Temporal Dynamics Encoder (MTDE), and optimized through Phase 0, Phase 1, and Phase 2, which are delimited by blue, orange, and green outlines, respectively. The arrows indicate the direction of the data flow and processing sequence.

• 

  Figure 2. Phase 0—exploration and representation learning. MTDE embeddings are projected into a 128-dimensional space and trained with Supervised Contrastive Loss, while an internal SoftMax attention head is regularized by entropy; Gated Pooling and the session classifier remain frozen. The arrows indicate the direction of the data flow and processing sequence.

• 

  Figure 3. Phase 1—chunk-level discrimination and attentional refinement. (a) Top-K α-Entmax attention selects the most informative chunks; (b) the chunk auxiliary classifier is trained with focal cross-entropy, and Gated Pooling parameters are unfrozen from epoch 25. The arrows indicate the direction of the data flow and processing sequence.

• 

  Figure 4. Phase 2—session-level inference and optimization. α-Entmax weights each chunk, Gated Pooling aggregates gated features into a 256-dimensional session vector, and the main classifier predicts high/low valence–arousal labels using session-level cross-entropy loss. The arrows indicate the direction of the data flow and processing sequence.