Multi-Stream Deepfake Video Detection Using Heartbeat Signals and Lip-Sync Verification

The rapid development of Artificial Intelligence (AI), deep learning, and generative models has enabled the creation of highly realistic manipulated videos known as deepfakes. These videos can alter facial identities, expressions, and lip movements, creating challenges for digital media authenticity and security. Conventional deepfake detection methods primarily rely on frame-level visual analysis, which may be insufficient when manipulation artifacts are subtle or videos undergo compression and post-processing. This paper proposes a multi-stream deepfake video detection system integrating Convolutional Neural Network–Long Short-Term Memory (CNN-LSTM), remote Photoplethysmography (rPPG), and lip-sync verification. The CNN-LSTM stream extracts spatial facial features and examines temporal inconsistencies across consecutive video frames. The rPPG stream analyzes subtle facial color variations to estimate physiological signals associated with blood-volume changes. The lip-sync verification stream examines the synchronization between audio and mouth movements. The outputs of these streams are intended to be integrated to support deepfake classification. By combining visual, temporal, physiological, and audio-visual evidence, the proposed system aims to provide a more comprehensive approach to deepfake video detection.