Multimodal Machine Translation for Children’s Picture Books: Combining Vision and NLP to Support Chinese-to English Bilingual Literacy in Primary Education

Multimodal machine translation is increasingly recognized as an effective tool for early bilingual literacy, especially for junior students engaging with picture-story books. The proposed Hybrid Transformer-ViT framework unifies text encoding and visual grounding, enabling coherent translations without redundant visuallinguistic features. Key components include advanced text preprocessing, image normalization, multimodal alignment, and crossattention fusion, enhanced by Target-Visual Consistency (TVC) and Bilingual-Visual Consistency (BiVC) mechanisms. Hyperparameters were optimized using the Grey Wolf Optimizer to accelerate convergence and improve robust cross-modal representation learning. Experiments on the 3AM dataset, containing 15,728 image-text pairs from 436 children’s books, achieved BLEU-4: 50.8, METEOR: 52.9, ROUGE-L: 64.1, and CIDEr: 1.73, outperforming the text-only Transformer (BLEU-4: 37.8). Human evaluation confirmed gains in fluency (+18.4%), adequacy (+30.6%), image-text consistency (+65.5%), and child readability (+31.4%). Optimized GPU deployment reduced latency to 27 ms, enabling real-time translation and enhanced bilingual story comprehension.