We present a doubly-attentive multimodal machine translation model.Our model learns to attend to source language and spatial-preserving CONV 5,4 visual features as separate attention mechanisms in a neural translation model.In image description translation experiments (Task 1), we find an improvement of 2.3 Meteor points compared to initialising the hidden state of the decoder with only the FC 7 features and 2.9 Meteor points compared to a text-only neural machine translation baseline, confirming the useful nature of attending to the CONV 5,4 features.
No takes yet. Share an insight, caveat, or question.
Calixto et al. (2016) studied this question.