Enhancing Cheapfake Detection: An Approach Using Prompt Engineering and Interleaved Text-Image Model

Misinformation is a widespread problem in the wake of flourishing social media. One of the most common forms of misinformation is cheapfakes - multimedia content that is manipulated with simple techniques such as miscaptioning to create misleading or false narratives. Due to their simplicity and appealing nature, cheapfakes pose a formidable threat to the reliability, transparency and integrity of the Internet. The ACM ICMR 2024 Grand Challenge on Detecting Cheapfakes tasks participants with detecting out-of-context cheapfakes - images with misleading or irrelevant captions. This paper introduces a lightweight approach that enhances cheapfake detection by combining Prompt Engineering with image captioning running on an interleaved image-text model. Testing on a public dataset gives an accuracy of 82.9%. Despite its modest accuracy, our method demonstrates the potential for applying better mixed-media learning models for context understanding and Visual Question Answering.