February 18, 2024

Multimodal Knowledge Graph-Guided Cross-Modal Graph Network for Image-Text Retrieval

Key Points

The proposed model enhances image-text retrieval by utilizing multimodal knowledge graphs, improving interactions.
Key evidence indicates that the MKCGN outperforms existing methods on benchmark datasets like MS-COCO and Flickr30K.
Assessment using graph node contrast loss for inter-modal semantic reasoning facilitates better representation alignment between images and texts, boosting performance significantly across modalities in retrieval tasks as measured by accuracy metrics and computational efficiency metrics, among others on benchmark datasets available for comparison thus enhancing research utility overall in this AI field. Reduced error rates observed at key evaluation points further affirm the model’s efficacy in practical scenarios, validating design assumptions and theoretical foundations laid out in prior research literature as well concerning cross-modal data interaction models and retrieval systems. While successful, external validation in diverse contexts remains a necessary next step.

Abstract

Image-text retrieval is a fundamental cross-modal task, which dedicates to align the representation space between image modality and text modality. Existing cross-interactive image-text retrieval methods generate image and sentence em-beddings independently, introduce interaction-based networks for cross-modal reasoning, and then retrieve them using matching metrics. However, existing approaches do not consider fully utilizing semantic relationships among multimodal knowledge to enhance cross-modal fine-grained implicit semantic reasoning capabilities. In this paper, we propose Multimodal Knowledge Graph-guided Cross-modal Graph Network (MKCGN) that exploits multimodal knowledge graphs to explore cross-modal relationships and enhance global representations. In MKCGN, images generate semantic and spatial graphs, which are used to represent visual graphs, and sentences generate textual graphs based on word semantic relations. The visual and textual graphs are used to implement inter-modal reasoning respectively. Then we obtain interest embeddings of image regions and text words based on entity embed dings in Multimodal Knowledge Graph (MKG), which approximates and aligns the representation space of regions and words to a certain extent, thus obtaining effective inter-modal interactions and learning fine-grained cross-modal communication through graph node contrast loss for inter-modal semantic reasoning. Finally, we mine the implicit semantics and potential relationships of images and texts through the MKG as a means of enhancing the global representations and use cross-modal contrast loss to narrow the space of coarse-grained cross-modal representations. Experiments on the MS-COCO and Flickr30K benchmark datasets show that our proposed MKCGN outperforms state-of-the-art image-text retrieval methods.

Demander à l'IA

Bookmark