Research on multimodal link prediction method based on Vision Transformer and convolutional neural network

Abstract To address the problems of inadequate feature representation and low generalisation ability of existing link prediction methods. A multimodal link prediction method based on Vision Transformer and convolutional neural network is proposed. Firstly, PHash is employed at the filter gate to filter out irrelevant images. Secondly, picture features are extracted using Vision Transformer model and computed using MRP through forgetting gate. Multi-layer Convolutional Neural Networks are used to fuse spatial location feature information during entity relationship embedding to effectively obtain richer semantic information. Meanwhile, a multi-scale null convolution kernel is used to capture rich explicit interaction features in different scale spaces. Finally, the picture features are fused with entity-relationship features in the fusion gate. The experimental results show that the MRR metrics are improved by 17.3% compared with the DistMult model on the public dataset and 4.1% compared with the TuckER model on the e-commerce dataset.