Answers are grounded in this paper and cite their sources.
Demonstrates improved accuracy in visual question answering using multimodal knowledge graphs.
In the domains of Natural Language Processing (NLP) and Computer Vision (CV) Visual Question Answering (VQA) is a multidisciplinary task, in which an image and a question are given to a VQA system, which is responsible for giving the answer. The VQA system is used for a variety of real-world applications, such as providing situational information based on visual material, making judgments using a vast quantity of surveillance data, interacting with robots, and helping individuals who are blind or visually impaired. Although it is required yet challenging to complete comprehensive VQA, Fact-based VQA (FVQA) approaches in which external knowledge is required to process with image and question. Existing FVQA methods combine all types of data without fine-grained selection, thereby generating unexpected noise while reasoning about the final result. The problem solution should be able to collect complementary-information evidence based on question-attention. We represent an image with different layers of information by a multimodal knowledge graph relating to the features of visual, factual, and semantic. We propose a multimodal knowledge graph-convolutional-network (GCN) to collect relevant-information evidence from different information layers based on the given question. In particular, intra-modal knowledge graph attention takes evidence from each modality, while inter-modal knowledge graph attention gets evidence across the different information layers. To get an optimal answer, we stack this process multiple times to perform a reasoning mechanism. Over the FVQA dataset, we achieved state-of-the-art results by improving 10.86% test accuracy, which demonstrates the usefulness and interpretability of our approach.
No takes yet. Share an insight, caveat, or question.
Hussain et al. (2022) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: