Answers are grounded in this paper and cite their sources.
Text-only neural machine translation (NMT) models can struggle with lexical, gender, and referential ambiguities when textual cues alone are insufficient. This paper presents a digital-driven deep-learning framework for text–image multimodal machine translation that integrates visual context through gated cross-modal fusion and contrastive text-image alignment. The architecture combines a frozen CLIP ViT-B/32 vision encoder with a trainable linear projection and a pretrained MarianMT text encoder-decoder. An adaptive sample-level scalar gate regulates the contribution of visual information according to cross-modal relevance. On the Multi30K EN–DE Test 2016 benchmark, the proposed framework achieves 40.9 BLEU and 0.867 COMET, improving upon text-only MarianMT by 2.2 BLEU and 0.025 COMET and outperforming the evaluated multimodal baselines across seven translation metrics. Paired bootstrap resampling with 1,000 iterations indicates statistically significant improvements ( \(p < 0.001\) , Cohen’s \(d = 0.72\) , 95% CI [1.2, 3.2]). Analysis of gate activations further examines how visual contribution varies across samples with different linguistic characteristics, without assuming token-level gating. The framework introduces an additional inference latency of 21.7 ms over the text-only baseline and is further evaluated under multiple visual degradation and image-relevance conditions. These results demonstrate the potential of adaptive gated fusion for incorporating visual evidence into multimodal machine translation while maintaining computational efficiency.
No takes yet. Share an insight, caveat, or question.
Nan Li (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: