Remote Blood Oxygen Estimation From Videos Using Neural Networks

Fig. 1.. The overall pipeline for our proposed method. The hand is first recorded and then the hand and skin pixels are segmented from the background. The average pixel values of the hand for each color channel is calculated frame by frame. This results in an RGB time series which is fed into the neural neural network model for SpO2 estimation.

Fig. 2.. Extinction coefficient curves of hemoglobin showing the absorption properties at different wavelengths. The curves were plotted based on [7], [18]. The difference between oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (Hb) at the red and blue/infrared wavelengths means that these color channels contain useful information for SpO2 prediction by means of optophysiological principles.

Fig. 3.. Proposed network structures for predicting an SpO2 level from a fixed-length segment of skin color signals. We highlight the differences among the three model configurations instead of showing the exact model structures. Model 1 combines the RGB channels before temporal feature extraction. Model 2 extracts the temporal features from each channel separately and fuses them toward the end. Model 3 interleaves color channel mixing and temporal feature extraction.

Fig. 4.. Fitzpatrick skin types [32].

Fig. 5.. Setup for capturing hand videos. Over the top is an smartphone device recording a single video for both hands. The left and right hands are in the palm down (PD) and palm up (PU) positions, respectively. The CMS-50E pulse oximeter clamped to the left index finger records reference SpO2 signals.

Fig. 6.. (a) Breathing protocol that participants were asked to follow, including 3 cycles of normal breathing and breath holding. (b) Histogram of SpO2 values in the collected dataset.

Fig. 7.. (a) Training vs. validation predictions. (b) Test predictions of varying performance with reference SpO2. The higher the Pearson’s correlation, the better the prediction captures the reference SpO2 trend. The lower the MAE, the better the prediction captures the dips in SpO2.

Fig. 8.. Cases where the proposed model fails to accurately predict SpO2. (a) The model predicts a near constant SpO2 that results in small MAE but also low correlation. (b) The model prediction follows the general trend of the reference SpO2 signal but does not capture the magnitude of the dips. This results in a high correlation but also large MAE.

Fig. 9.. Box plots comparing distributions of correlations for (a) lighter vs. darker skin types, and (b) PD vs. PU for all skin types. The PD results are better for darker skin tones in both the participant-specific and leave-one-out cases.

Fig. 10.. Posterior distribution of the difference of group means. This shows an example of an undecided case of the Bayesian statistical test given that the ROPE of zero difference is set to [−0.03, 0.03] and 33% of the posterior distribution falls within the ROPE. The percentage of coverage can be used to quantify the certainty that two groups have the same mean.

Fig. 11.. Histograms of correlation values between reference SpO2 signals and (a) randomly generated SpO2 signals, or (b) SpO2 signals predicted by neural network Model 2. The correlation distribution for Model 2 is centered much higher than the random guess, confirming Model 2’s capability to track SpO2.

Fig. 12.. Learned RGB channel weights. Plots (a) and (b) are the channel weights learned by different model instances trained on the data of all study participants together, projected onto the RB and RG planes in the RGB space. Plots (c) and (d) are the RB and RG projections of the learned channel weights for model instances trained on random subsets of the participants’ data. Each point is color-coded according to the correlation ρ achieved by the instance.