COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning

Despite the recent attention to DeepFakes, one of the most prevalent ways to audiences on social media is the use of unaltered images in a new but context. To address these challenges and support fact-checkers, we a new method that automatically detects out-of-context image and text. Our key insight is to leverage the grounding of image with text to out-of-context scenarios that cannot be disambiguated with language. We propose a self-supervised training strategy where we only need a set captioned images. At train time, our method learns to selectively align objects in an image with textual claims, without explicit. At test time, we check if both captions correspond to the same (s) in the image but are semantically different, which allows us to make accurate out-of-context predictions. Our method achieves 85%-of-context detection accuracy. To facilitate benchmarking of this task, we a large-scale dataset of 200K images with 450K textual captions from a of news websites, blogs, and social media posts. The dataset and source is publicly available at: //shivangi-aneja. github. io/projects/cosmos/.