Semantic consistency-aware multimodal deep learning for misinformation detection in short-form videos

The use of short-form videos, or social media "vines", has led to a dramatic increase in the rate of misinformation sharing, which presents a major threat to society. The challenge of detecting this type of content is that it is multimodal and, therefore, the text, video and audio cues may present subtle inconsistencies. Although previous multimodal approaches mainly focus on feature fusion, they tend to ignore the semantic inconsistencies across modalities. To address this, this paper proposes a semantic consistency-aware multimodal deep learning framework to detect misinformation in short videos. This approach involves modality-specific feature extraction, attention-based feature fusion and a semantic consistency module that measures the semantic consistency across modalities. The consistency score is used as an additional feature for the final decision, enabling the model to learn from inconsistencies within the modalities. We evaluate the proposed model on typical multimodal datasets such as SocialVidMix using popular performance measures. Our approach outperforms the conventional unimodal and multimodal fusion methods, suggesting that semantic consistency is an important addition to multimodal learning. An ablation study also verifies the effectiveness of the consistency component in enhancing detection accuracy. While the existing approach focuses on semantic alignment in the representation space and does not explicitly incorporate temporal information, the results suggest that even a simple consistency term offers discriminative power. This model will look to more complex consistency modeling and temporal modeling in future work.