A Graph-Based Fusion Model Propagating Multimodal Information for Recommendation

Abstract Multimedia data is inherently multimodal, encompassing information across various modalities like text, image, and audio. Each of these modalities represents a distinct method or perspective, forming a rich tapestry of data. In contrast to the structured data found in conventional databases, multimedia data presents distinctive features such as high dimensionality, content richness, and substantial storage requirements. Existing multimodal recommendation systems struggle to achieve rational results. The contributions of our paper are as follows. The traditional approach to representing multimodal knowledge graphs focuses solely on training embedding vectors through triples, neglecting the crucial process of fusing information from displayed multimodalities. This paper tackles this issue by ingeniously transforming the multimodal information fusion process in knowledge graphs into a graph-based information propagation process, introducing a novel model. Multimodal labeling, building on unimodal labeling, is introduced to characterize the multimodal semantic information within the video. Additionally, a multicore learning architecture is employed to amalgamate the semantics of multiple modalities, enabling the mapping of underlying multimodal features to a unified intermediate semantic space. This facilitates learning a similarity metric function between videos based on the intermediate semantic space. Extensive experiments are conducted on real datasets in this paper. The results demonstrate that our model surpasses current mainstream collaborative filtering-based recommendation algorithms, knowledge graph-based recommendation algorithms, and multimodal learning-based recommendation algorithms.