Most question answering systems use symbolic or text information. We present a dataset for a task that requires understanding descriptions of visual themes and their layout: identifying paintings from their descriptions. We annotate paintings with contour data, align regions with entity mentions from an ontology, and associate image regions with text spans from descriptions. A simple embedding-based method applied to text-to-image coreferences achieves state-of-the-art results on our task when paired with bipartite matching. The task is made all the more difficult by scarcity of training data.
No takes yet. Share an insight, caveat, or question.
Guha et al. (2016) studied this question.