Text-based image captioning (TextCaptioning) aims to depict an image by comprehending the scene texts and other visual cues it encompasses. To endow captioning models with text-reading capabilities, a common strategy is to leverage external OCR systems. However, OCR tokens produced by these systems often exhibit inaccuracies in real-world scenarios, posing significant challenges for TextCaptioning. These inaccuracies can introduce disparities between scene texts in images and words in sentences, thereby undermining the carefully established semantic consistency between vision and language in captioning models. To tackle this challenge, we introduce the Text-aware Self-Critical Training (TSCT) method, which leverages a text-aware reward along with the REINFORCE algorithm to optimize TextCaptioning models. By providing appropriate rewards for individual words, TSCT enables more refined model optimization and helps mitigate the negative impacts of unreliable OCR tokens. Additionally, we enhance the language model by integrating token type information of input words, which is obtained by collecting a sentence dataset based upon the original TextCaps dataset. This additional information helps alleviate the ambiguity in matching sentence words with the standard vocabulary or scene texts caused by unreliable OCR tokens, thereby improving the quality of learned word embeddings and OCR token embeddings. Experimental results demonstrate that our proposed method outperforms existing state-of-the-art approaches, leading to a significant enhancement of CIDEr-D from 117.1% to 132.6%.
Wang et al. (Thu,) studied this question.