OCR Based Deep Learning Approach for Image Captioning

The task of automatically describing photos based on reading and understanding textual text present in photographs is known as OCR-based image captioning. When compared to traditional captioning, OCR-based picture captioning is a difficult task, especially when images contain several text tokens and visual components. Using textual information for sentence synthesis and predicting text tokens are two challenges that have not been thoroughly studied in previous research. This motivates us to do the research in this direction. So this research paper aims to generate a semantically enriched description of an image. Our suggested method uses Clip-GPT2 for the caption generation model and OCR techniques for extracting text tokens from images and the sentence fusion model BART for text summarization. Extensive experiment has been done on the Flickr30K and T2SUM datasets to verify the efficacy of our proposed approach. As a result, our approach assists visually challenged users in communicating the contextual information of an image through an insightful textual description.