Contactless camera-based estimation of heart rate and variability from short facial video using remote photoplethysmography, remote ballistocardiography, and quality-aware selection

Background Contactless vital-sign monitoring from ordinary cameras can expand access to physiological assessment in telehealth and consumer devices. A key limitation is variability in signal quality driven by illumination, motion, and skin phototype, which can reduce robustness and fairness. Purpose To describe and evaluate a camera-based technology that estimates heart rate and heart rate variability from 30–60 s facial video by combining remote photoplethysmography with remote ballistocardiography and selecting the better modality per time window based on signal quality. Methods The analytic sample comprised N = 5,311 participants (64.7% women, 35.3% men), with mean age 53.8 ± 18.4 years. Fitzpatrick classification was available for 99.9% (5,306/5,311): Type IV 57.8% (3,068), Type III 27.6% (1,465), Type V 11.2% (593), Type VI 3.2% (169), and Types I–II 0.1% each. The system performs end-to-end video-based signal extraction, signal-quality assessment, and prediction. Remote photoplethysmography uses subtle facial color changes associated with blood volume pulse, while remote ballistocardiography uses minute facial micro-movements linked to cardiac mechanical activity. A signal-quality–aware selector chooses the modality expected to be more reliable for each analysis window. Results In internal testing, remote photoplethysmography achieved heart rate mean absolute error of 0.37 beats per minute and heart rate variability (standard deviation of normal-to-normal intervals) mean absolute error of 5.85 milliseconds; at high signal quality, errors decreased to 0.07 beats per minute and 2.00 milliseconds. Remote ballistocardiography was less accurate in aggregate (heart rate mean absolute error 3.57 beats per minute; standard deviation of normal-to-normal intervals mean absolute error 30.59 milliseconds) but was invoked in approximately 4% of windows to improve robustness. Modality selection reduced the final standard deviation of normal-to-normal intervals error to 5.876 milliseconds (about 7% improvement versus remote photoplethysmography alone) and improved performance for darker skin tones, including about 18% improvement for Fitzpatrick Type VI. Conclusions Combining remote photoplethysmography with complementary remote ballistocardiography and signal-quality–aware modality selection enables robust, camera-based estimation of heart rate and heart rate variability from short facial video, with measurable gains in challenging conditions and improved performance for darker phototypes.