Cross-Modal Tri-Semantic Correlation-CLIP for Short Video Homogenization Recognition

Short videos are one of the most popular social media in the world, triggering a proliferation of copycat creations leading to homogenized video content, with visual and textual homogenization being the most prevalent. Unlike near-duplicate video retrieval, which relies on visual appearance similarity, homogenization recognition emphasizes identifying videos with similar semantic units. Short videos exhibit multimodal features, in which there is a many-to-many mapping relationship between visual and text elements, and the two modalities are relatively independent and semantically correlated. Therefore, cross-modal semantic correlation needs to be explored and established to achieve homogenization recognition of short videos. Based on the idea of divide-and-conquer and joint processing, we propose a cross-modal tri-semantic correlation-CLIP (CS 3 C-CLIP) for short video homogenization recognition. First, visual and text features in the shared subspace are extracted using the contrastive language-image pre-training visual-text dual encoder. Then, features at the patch, frame, and video levels are generated using the patch selection module and the temporal encoder, while the word-level and sentence-level features are respectively derived from text features and [EOS] token. After establishing cross-modal tri-semantic correlations by constructing a triple semantic (i.e., video-sentence, frame-sentence and patch-word) correlation, homogenized short videos are recognized by measuring the aggregated cross-modal similarity between pairs of short videos. Experimental results on three publicly available datasets demonstrate that our CS 3 C-CLIP outperforms state-of-the-art methods, achieving 85.7% R@1 and 94.4% R@5 on self-built BJUT-HCD, 49.4% R@1 and 74.6% R@5 on MSR-VTT, and 49.8% R@1 and 78.1% R@5 on MSVD, respectively.