Efficient Spatiotemporal Attention Network for Remote Heart Rate Variability Analysis

Figures

• 

  Figure 1. Signal waveform obtained by ECG and cPPG technology.

• 

  Figure 2. Proposed HRV analysis framework using ESA-rPPGNet.

• 

  Figure 3. Normal-to-normal interval (NNI) obtained from rPPG signal.

• 

  Figure 4. Comparison of 2D (a) and 3D (b) convolutions. In 3D convolution, the same 3D kernel is applied to overlapping 3D cubes in the input video to extract motion features.

• 

  Figure 5. Face detection using MTCNN algorithm.

• 

  Figure 6. The process of standard 3D convolution.

• 

  Figure 7. The process of 3D depth-wise convolution.

• 

  Figure 8. The process of 3D point-wise convolution.

• 

  Figure 9. An overview of the proposed SA module. The C channels are divided into G groups and each group has C/G channels. “Split” means “channel split”. It adopts “channel split” to process the sub-features of each group in parallel. Each group is divided into two sub-features and each sub-feature has C/2G channels. Fgp means to use global average pooling (GAP) to generate channel-wise statistics; then, Fc uses a pair of parameters to scale and shift the channel vector. σ means to use the sigmoid activation function to create a compact feature to guide more precise and adaptive channel selection. Group norm (GN) is adopted to generate spatial-wise statistics. “Fuse” means keep the tensor unchanged, directly as input. The two branches are then concatenated. After that, all sub-features are aggregated, and finally, SA utilizes a “channel shuffle” operator to enable information communication between different sub-features.

• 

  Figure 10. Structure of ESA-rPPGNet. ESA is a spatiotemporal feature extraction network composed of 3DCNN. ConvGRU is a recurrent convolutional neural network proposed in Chapter 3. Here, 3D AvgPooling is used to reduce the spatial dimension of feature map to one. Conv3d is a 3D convolution used to obtain the final rPPG signal.

• 

  Figure 11. The basic blocks of ESA-rPPGNet. (a) is the basic component block of the encoder. (b) is the basic component block of the decoder.

• 

  Figure 12. Example images from the two datasets used in this paper.