Multimodal Audio-Visual Deepfake Detection: Validity Checks Catch a Detector That Looks Right and Is Not

Lip-sync deepfakes manipulate the mouth region and the accompanying voice while leaving most of the original video untouched. This creates a problem for detectors that rely only on visual information, since much of the footage they analyse may still be genuine. Combining audio and video seems like a natural solution, which led us to develop a system with separate visual and audio streams and two cross-attention streams designed to compare mouth movements with speech and facial information with voice characteristics. Their outputs are combined at the feature level. Rather than assuming that these components work as intended, however, we examined whether their behaviour supports that assumption. We asked whether cue-specific fusion could generalize better than a strong visual baseline under source-disjoint evaluation and controlled testing for shortcut learning. Our results did not support this expectation. Fusion improved the in-domain ROC-AUC by 0.0159, but it failed to outperform the visual baseline on an unseen corpus across all three seeds and performed significantly worse for two of them. The experiments also exposed problems that would have been easy to overlook if we had considered accuracy alone. One cross-attention stream achieved a ROC-AUC of 0.9991 while its attention weights remained almost uniform throughout training, suggesting that the mechanism it was designed around was not contributing as intended. We also found evidence of codec-related shortcut learning: when genuine clips were re-encoded using the codec associated with the largest fake category, 93.3% were classified as fake. Combining compression augmentation with codec-swapped checkpoint selection reduced cross-corpus ROC-AUC by 0.19. In another experiment, selecting a checkpoint using validation loss resulted in below-chance performance, while changing only the training seed produced a cross-corpus ROC-AUC spread of 0.1078. These findings show why it is important to monitor properties that are not directly optimized during training. We present our evaluation protocol, the measurements collected, and the failures identified throughout the study.