Reliable multimodal signals for in-vehicle heartbeat detection

Continuous and unobtrusive cardiac monitoring during driving is medically attractive but technically challenging due to motion artifacts, contact variability, and changing illumination. Building on our previously developed multimodal sensing platform that integrates steering wheel electrocardiography (ECG), photoplethysmography (PPG), and a camera (RGB), in this paper, we (i) propose an improved electrode design for steering wheel ECG acquisition, (ii) integrate infrared and thermal cameras, and (iii) combine these modalities for robust heartbeat detection under real-world driving conditions. We record data from 23 healthy adults on a standardized 55 km route that includes urban, rural, and highway segments. During each drive, we acquire a ground truth ECG using electrodes attached to the driver’s upper body. We use convolutional neural network models with early fusion, signal-based late fusion, sensor-based late fusion, and hybrid fusion. The steering wheel ECG achieves the highest single-modality performance (F1-score = 91.92%). Augmenting ECG with iPPG-RGB and iPPG-IR raises the F1-score to 92.14%. The vision-based signals provide limited complementary information but do not compensate for complete steering wheel electrocardiogram (ECG) loss. In future work, we will increase the reliability of physiological recordings by using data from the vehicle’s controller area network (CAN) bus.