Passive heart-rate monitoring during smartphone use in everyday life

Figures

• 

  Fig. 1. Overview, development and validation of the PHRM system. a, In our research study with consented participants, after a screen-unlock event, PHRM passively captures, processes and analyses 8-s facial video clips using a deep neural network (DNN) to estimate HR and associated prediction confidence to determine whether the measurement is valid. To compute daily RHR, PHRM aggregates valid HR measurements from intermittent 8-s video clips throughout a single day and applies a Kalman filter to improve estimates. b, Workflow diagram of the studies used to develop and validate the PHRM system. We used data from five independent, prospective laboratory studies and a prospective free-living study.

• 

  Fig. 2. Representative examples of the diversity of free-living data used to validate the PHRM system. a, Illustrative examples of the variety of environments, lighting conditions, front-facing camera angles and face obstructions for videos captured in the free-living conditions. b, Examples of facial skin patches randomly sampled from video frames of the cheeks of participants across the full range of MST values. Videos are sorted by mean brightness across columns and MST across rows. c, From left to right: histograms of the number of 8-s video clips by time of day; illuminance measured by the smartphone ambient light sensor; and the average magnitude of linear acceleration of the smartphone during the videos.

• 

  Fig. 3. Comparison of PHRM and state-of-the-art rPPG methods on our HR test benchmarks. a,b, MAPE values at the participant level of PHRM and 15 baselines by skin-tone group in diverse laboratory (a; n = 44, 25 and 35 participants in the light, medium and dark skin-tone groups, respectively) and free-living (b; n = 39, 29 and 39 participants in the light, medium and dark skin-tone groups, respectively) conditions. Deep-learning baseline models were trained on the PURE (ending with -P) and RLAP (ending with -R) datasets. Filled bars indicate MAPE on valid HR measurements after gating on the confidence scores associated with the PHRM predictions; empty bars indicate MAPE on all videos without any gating. Red dashed lines indicate the prespecified accuracy target of MAPE <10% according to the ANSI/CTA-2065 standard. Error bars are upper 95% CIs.

• 

  Fig. 4. Accuracy of passive HR and RHR measurements by PHRM in free-living conditions. a, Bland–Altman plot showing the agreement between PHRM-estimated HR values and reference ECG measurements. Colours indicate the confidence level of PHRM predictions. Dashed lines show the bias, lower and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. b, Box plots showing the distribution of MAPE values for individual participants, grouped by skin pigmentation. n = 39, 29 and 39 participants in skin-tone groups 1, 2 and 3, respectively. The box bounds the IQR divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. Red dashed line indicates the prespecified accuracy target of MAPE <10%. c, Bland–Altman plot showing the agreement between PHRM-estimated daily RHR values and the reference wearable HR tracker measurements. Colours indicate the day number since the start of RHR predictions. Dashed lines show the bias, lower and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. d, MAE of PHRM-estimated RHR as a function of day number since the start of RHR predictions, grouped by skin…

• 

  Extended Data Fig. 1. Accuracy of PHRM HR measurements in laboratory settings. a, Bland–Altman plot showing the agreement between PHRM-estimated HR values and the reference ECG measurements. Colours indicate the confidence level of PHRM predictions. Dashed lines show the bias, lower, and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. b, Box plots showing the distribution of MAPE values for individual participants, grouped by skin pigmentation. The box bounds the IQR divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. The red dashed line indicates the prespecified accuracy target of MAPE < 10%. c, Boxplots showing the distribution of mean absolute error (MAE) values for individual participants, grouped by skin pigmentation. The box bounds the IQR divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box.

• 

  Extended Data Fig. 2. Participant-level rates of video approval and upload. The distribution of approved videos per participant was strongly left-skewed, indicating that most participants uploaded a high proportion of their videos.

• 

  Extended Data Fig. 3. Comparison of daily RHR measurements by PHRM and reference and conventional RHR measurements in free-living conditions. a–c, Scatter plots showing the agreement between PHRM-estimated daily RHR values and reference RHR from a Fitbit wearable HR tracker (a), supine RHR measurements from ECG (b) and sitting RHR measurements from ECG (c), respectively.

• 

  Extended Data Fig. 4. Trends in daily RHR measurements over the week. Comparison of daily RHR estimates from the proposed PHRM system and a reference wearable RHR tracker over a seven-day period, sampled from participants exhibiting diverse trends. To enhance interpretability, participants were categorized into three groups on the basis of reference wearable RHR trends: decreasing, increasing and other patterns.