Multimodal scene-graph matching for cheapfakes detection

Abstract The development of technology and social media platforms has led to the proliferation of fake news, including the cheapfakes problem. Cheapfakes can be produced easily and spread quickly; a common type is out-of-context misinformation. It is worth investigating a proper algorithm that can efficiently detect such types of misinformation. In this work, we study a new approach to the detection problem of cheapfakes by using graphical neural networks to detect out-of-context samples using the dataset from the ICME 23 Grand Challenge on Detecting Cheapfakes. Specifically, our model utilizes scene graph matching and a language model pre-trained for the natural language inference task to solve task 1 of the challenge. We also propose effective methods to generate labeled data from an unlabeled training set of the challenge. Our proposed method achieved an F1-score of 85.69% and an accuracy score of 85.50% on the public testing set, surpassing the baseline by 4.69% and 3.60%, respectively.