TDA-Phys: Temporal Difference Adaptation of Video Foundation Model for Remote Photoplethysmography

Remote photoplethysmography (rPPG) enables noncontact estimation of vital signs, particularly heart rate, by analyzing subtle periodic skin color variations in facial videos. While deep learning has advanced rPPG signal extraction, existing methods rely on carefully designed task-specific architectures that are costly to develop and generalize poorly. In this work, we demonstrate that the general video foundation model VideoMAE v2 can be effectively adapted to the rPPG signal regression task by introducing only a lightweight adapter, without modifying its pretrained backbone. We freeze the entire VideoMAE v2 encoder and introduce a Temporal Difference Convolutional Adapter to capture the subtle interframe intensity differences. To address the mismatch between VideoMAE v2′s short input window (16 frames) and the long temporal context typically required for robust rPPG extraction (e.g., 160 frames), we adopt an overlapping sliding window strategy for segmented inference and reconstruct the full signal through weighted temporal aggregation. On the COHFACE and UBFC-rPPG datasets, our method achieves mean absolute errors (MAEs) of 0.90 and 1.55, reducing the error by more than 55% and 42%, respectively, compared to PhysFormer (2.00 and 2.70). Furthermore, on challenging real-world datasets such as BUAA-MIHR, which features strong illumination variations, and VIPL-HR, which involves significant head movements, our approach achieves MAEs of 6.68 and 8.23, respectively, despite incorporating no task-specific robustness modules. These results demonstrate stable rPPG signal recovery and validate the feasibility of leveraging general video foundation models for physiological signal perception.