Integrating perceptual quality analysis and caption-based features for robust deepfake video detection

This is the authors' abstract. We don't add key points for this paper.

Inspired by the benefits of both linguistic representations as well as visual quality analysis, we propose a multimodal framework for deepfake detection with reliable accuracy . Our method acquires low-level perceptual features from frames of video, including contrast, brightness, and sharpness, and computes artifact scores for artificial anomaly detection. In parallel, our method produces descriptive frame-level captions, summed up for generating video-level summaries for capturing contextual coherence. For training the model, our method leverages the FaceForensics++ dataset, comprising several techniques of deepFake manipulation, including DeepFake, Face2Face, FaceSwap, as well as NeuralTextures. Metadata comprised of quality measurements, the artifact scores, as well as textual captions, is tokenized, processed by a DeepSeek V2 Lite model , fine-tuned by the Low-Rank Adaptation (LoRA) procedure, as the backbone for classification. We test the proposed model on both intra-dataset as well as cross-dataset settings, demonstrating good accuracy as well as impressive generalization ability. Our method achieves a 96.51% accuracy for classifying, by integrating low-level perceptual as well as high-level semantic reason, effectively marking manipulated content as well as upholding the integrity of the medium in a world of increasingly evolving generative methods. The employed source code as well as dataset preprocessing scripts for our work are made available for public access on: https://github.com/HessenUPHF/perceptual-caption-detection . • To identify deepfake videos, we propose a multimodal approach that combines semantic captioning with perceptual quality metrics. • We propose a multimodal method combining captions with perceptual metrics. • Video frames are analyzed using brightness, contrast, sharpness, and artifacts. • Captions and perceptual metadata are merged for robust multimodal classification. • Our LoRA-tuned DeepSeek V2 Lite model achieves 96.51% accuracy with few parameters.