Representation and Fusion Based on Knowledge Graph in Multi-Modal Semantic Communication

The existing research on multi-modal semantic communication ignores the exploration of reasoning correlation among multi-modal data. Motivated by this, a multi-modal semantic representation and fusion model based on knowledge graph (KG-MSF) is proposed in this paper. In KG-MSF, the direct and reasoning correlation semantic information is extracted and mapped into a two-layer semantic architecture to represent the semantics of each modal fully. After that, the knowledge graph with structural advantage is utilized to fuse multi-modal semantic information, which is transmitted under different channel conditions. To assess the efficacy of semantic representation and fusion of the proposed KG-MSF in the multi-modal semantic communication system, we conduct comprehensive experiments on the task of visual question answer (VQA) with a metric of answer accuracy. The results demonstrate the superiority compared with existing models for multi-modal semantic representation, fusion, transmission efficiency and channel robustness.