Preserving privacy and video quality through remote physiological signal removal

Figures

• 

  Fig. 1. Overview of the remote photoplethysmography (rPPG) estimation process. This figure illustrates the workflow of rPPG-based heart rate estimation, from video input to beats per minute (BPM) output. The process includes region of interest (ROI) processing, red, green, blue (RGB) computation, RGB preprocessing, blood volume pulse (BVP) signal estimation, and heart rate extraction using power spectral density (PSD) analysis.

• 

  Fig. 2. Overview of the frame modification pipeline. This figure outlines the three-step process of frame modification: region of interest (ROI) extraction, ROI modification, and ROI combination. The pipeline illustrates the application of 11 different modification techniques to three distinct ROIs: full frame, face excluding eyes and mouth, and forehead and cheeks only. The process begins with ROI extraction from a frame, followed by the application of modification techniques, and concludes with the modified ROI being overlaid back onto the original frame. Techniques requiring multiple frames for modification are denoted with asterisks.

• 

  Fig. 3. Impact of frame modification methods in heart rate estimation accuracy. This figure presents the average absolute difference in beats per minute (∣Δbpm∣) for various frame modification methods across different rPPG estimation techniques. The box and whisker plots represent the data distribution, where the median is shown by the horizontal black line, the colored boxes indicate the first and third quartiles and the whiskers extend from the box to the farthest data point within 1.5 times the inter-quartile range from the box. Outlier points are those lying past the end of the whiskers. The facial region excluding eyes and mouth was used as the region of interest (ROI). The results are estimated for each subject (n = 6) and each the activity (n = 4) in the LGI-PPGI dataset, averaged for all rPPG estimation techniques (n = 5) and over the entire length of the videos. All statistical values (means and quartiles) were computed using Python’s matplotlib package. Source data provided in Supplementary Table 3.

• 

  Fig. 4. Performance evaluation of frame modification methods. a Mean squared error (MSE) quantifying data loss between original and modified frames. b Processing speed in frames per second (fps) for each modification method. Both metrics were evaluated using the facial region excluding mouth and eyes as the region of interest (ROI). The MSE value was computed by averaging for all activities (n = 4) and subjects (n = 6), while fps was estimated by running an analysis on a MacBook Pro with Intel i5 and was averaged over 1000 frames. Source data for (b) provided in Supplementary Table 4.

• 

  Fig. 5. Heatmap of overall evaluation scores for frame modification methods and rPPG estimation techniques. This figure presents the overall evaluation score (OS) for each combination of modification method and remote photoplethysmography (rPPG) estimation technique. OS combines the average absolute difference in beats per minute and the mean squared error between frames to assess the complete performance of modification methods. AVG represents the average across all rPPG estimation techniques (n = 5). The color bar ranges from 0.2 to 0.75, with higher scores (lighter colors) indicating better performance. The results are averaged for all the subjects (n = 6) in the LGI-PPGI dataset and for all the activities (n = 4) over the entire length of the videos.

• 

  Fig. 6. Overall evaluation scores of video modification methods across different activities. This figure presents the overall evaluation score (OS) for each video modification method, divided by activity: (a) resting, (b) rotation, (c) talk, and (d) gym. The OS is represented by the full length of the bar, while the blue portion represents the contribution of the normalized error in beats per minute (∣Δbpmn∣) to the OS and the red portion the contribution of the normalized data retention factor (1 − MSEn) to the OS. The results are averaged for all the subjects (n = 6) in the LGI-PPGI dataset and for all the rPPG estimation techniques (n=5) over the entire length of the videos.

• 

  Fig. 7. Comparison of three regions of interest (ROIs) for frame modification. This figure illustrates the evaluation of three ROIs in terms of: (a) average ∣Δbpm∣, (b) average mean squared error (MSE), and (c) Overall evaluation score (OS). The results are averaged for one the subjects (n=1) in the LGI-PPGI dataset, for all the rPPG estimation techniques (n=5) and for all the activities (n=4) over the entire length of the videos. Source data is provided in Supplementary Table 5.