Noncontact Heart Rate Estimation via Space–Time Video Super-Resolution for rPPG Information Recovery

Heart rate is a vital physiological indicator that can be estimated in a non-contact manner through remote photoplethysmography (rPPG) using facial videos. Video quality is essential for the extraction of the rPPG signal and the accurate estimation of heart rate. Insufficient frame rate and resolution can cause loss of rPPG information, compromising heart rate estimation accuracy. To address this problem, a two-stage non-contact heart rate estimation method is proposed, which incorporates space-time video super-resolution (STVSR) technology for low-frame-rate and low-resolution video scenes. In the first stage, an STVSR network (FrameReconNet) and an rPPG information recovery network (rPPGRecovNet) are cascaded to achieve simultaneous video reconstruction and rPPG information recovery, guided by a hybrid loss function. In the second stage, an end-to-end network (rPPGExtractNet) is used to extract rPPG signals from videos reconstructed by FrameReconNet and estimate the final heart rate value. To address the limitation of the original STVSR network, which extracts only local features from short frame sequences and leads to rPPG discontinuity between adjacent sequences, a novel local-global-local (LGL) temporal feature aggregation and partition operation is proposed. This ensures consistent global temporal information across all frame sequences in a facial video. The experimental results on the VIPL and MMVS datasets demonstrate that the proposed method is an effective non-contact heart rate estimation method, which can recover rPPG information from low-frame-rate and low-resolution facial videos. The proposed method achieves comparable heart rate estimation accuracy with the fewest parameters, balancing the requirements of accuracy and real-time performance.