Improving Data Fusion for Fake News Detection: A Hybrid Fusion Approach for Unimodal and Multimodal Data

The proliferation of fake news, exacerbated by social media and modern technology, presents significant challenges across sectors such as health, the economy, politics, and national stability. This study addresses the limitations of current multimodal fake news detection models, which often struggle to effectively integrate heterogeneous modalities like text and images. We propose a hybrid data fusion approach (HF-TIM) that combines the early fusion of multimodal data with the late fusion of unimodal data, leveraging the strengths of both techniques to enhance detection accuracy. Our HF-TIM approach employs a Softmax classifier for early fusion and integrates it with unimodal features extracted from BERT and VGG-19 classifiers through a neural network-based meta-learning classifier. This approach captures the complementary and unique properties of each modality, resulting in a more comprehensive and robust fake news detection model. Experimental results demonstrate that the HF-TIM method significantly improves classification accuracy across various fake news categories by effectively addressing the complex interrelationships between text and images. Our fine-grained detection model, based on the HF-TIM method, achieved a detection accuracy of 93.4%, outperforming state-of-the-art models in related studies. The proposed hybrid fusion HF-TIM approach offers an innovative and effective solution for multimodal fake news detection, with potential applications extending to other domains.