Tri-FusionNet: Enhancing image description generation with transformer-based fusion network and dual attention mechanism

Captioning of images is a crucial step towards helping artificial intelligence to perceive images and create relatable language descriptions. Nevertheless, most current captioning models are not very effective at integrating fine-grained visual information with global contextual information, also leading to high correspondence between visual and text representations. We suggest Tri-FusionNet to resolve these difficulties, a new transformer-based image description generation model. The proposed model comprises a Vision Transformer encoder, a dual-attention mechanism, a RoBERTa-based decoder, and a CLIP-based integration model. The dual attentions of the visual encoder are performed to obtain both spatial and contextual information, which enhances the feature construct of images, whereas the decoder of RoBERTa produces coherent and correct text descriptions. The CLIP integration module matches visual and textual modalities by contrastive learning, which allows learning multimodal representations to occur. The Flickr30k, Flickr8k, and MS-COCO datasets have been evaluated experimentally to show that the proposed framework is effective. Tri-FusionNet attained the BLEU scores between 0.767 and 0.456, CIDEr scores between 1.679 and 1.483, METEOR scores between 0.478 and 0.358, and ROUGEL scores between 0.567 and 0.789, respectively, on Flickr30k and Flickr8k. The model scored 0.893 (B-1), 0.821 (B-2), 0.794 (B-3), and 0.725 (B-4) on MS-COCO. These findings indicate that Tri-FusionNet is highly effective in enhancing multimodal features fusion and provides more accurate and contextually relevant image descriptions.