Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement

Figures

• 

  FIGURE 1. The schematic diagram of rPPG principle. The absorption and reflection of light by the skin varies according to the hemodynamic status under light sources, such as sunlight or lamps. Such changes will be recorded by imaging devices (including industrial cameras, webcams, cell phone lenses and other imaging devices) in the form of videos or pictures. Through the processing of computer and algorithm, rPPG waveforms that represent physiological information can be obtained from these videos.

• 

  FIGURE 2. An ideal pipeline for contactless monitoring of flap blood supply. Firstly, the DL-based method is applied to original videos or images to realize super-resolution. Then the DL-based method is used to accomplish the preprocessing steps such as ROI selection, segmentation and tracking. Additionally, RGB signal can be converted into a HSV model with more detailed information. And the raw rPPG signal is obtained through a series of algorithms. Finally, the DL-based method is used to process the raw rPPG signal to obtain accurate physiological information.

• 

  FIGURE 3. Contactless HR estimation pipeline based on videos. The contactless HR estimation pipeline is composed of face video processing, face BVP signal extraction and HR computation. Face video processing includes video super-resolution, face detection and tracking, skin segmentation, and ROI selection. BVP signal extraction includes the filter denoising methods for motion artifacts filtering and skin color normalization and the conventional algorithms for raw BVP signals construction. HR computation methods are divided into time domain analysis and frequency domain analysis. DL algorithms can be divided into end-to-end type and hybrid type. The former directly establish the mapping from video frames to the target HR values or BVP signals, while the latter use DL model in conjunction with traditional ML methods or different DL models to deal with different stages of HR estimation. The face image in the schematic diagram comes from Chicago Face Database (Ma et al., 2015).

• 

  FIGURE 4. Supervised and self-supervised learning in rPPG signal prediction. A video clip is sampled from the source video first, then passed through the saliency sampler to generate the warped anchor. The anchor is passed through a PPG Estimator to get rPPG signal. If supervised training is employed, we employ a maximum cross-correlation (MCC) loss between the ground truth (cPPG) and predicted rPPG signal. If contrastive training is used, a random frequency ratio is sampled from a prior distribution. The warped clip is passed through the frequency resample to produce the negative sample, showing a subject with an artificially higher heart rate. This sample is passed through to produce the negative example PPG. The negative sample is again resampled with the inverse of random frequency ratio to produce a positive example PPG. Finally, the contrastive loss, multi-view triplet loss, is applied to the PPG samples, using a PSE MSE distance metric. The face images in the schematic diagram come from the Chicago Face Database (Ma et al., 2015).