Trustworthy heuristic graph learning for image-text matching

With decades of continuous research, existing image-text matching methods can satisfactorily achieve visual language alignment. However, new challenges are also emerging: after obtaining initial semantic-matching responses, the intrinsic causality and the massive data relationships have gradually attracted attention. Besides, with the development of large language models based on supercomputing resources, leveraging other effective designs and highlighting the superiority of one’s own research is also worth considering. Therefore, this article reflects on the future development of image-text matching, proposes three factors that need to be emphasized, i.e., plug-and-play, causality, and graph representation, then designs Trustworthy Heuristic Graph Learning (THGL for short) for image-text matching. Specifically, for the image-text pairs in the dataset, we first map them to the common graph representation space. Second, in each batch training, heuristic graph learning can improve the efficiency and causality of cross-modal semantic mining. Finally, cross-modal alignment integrates previous bidirectional learning to improve the overall performance of the detachable module. Besides, graph pooling reorders the patch information in feature computation, combining the advantages of maximum pooling and average pooling. Extensive experiments evidence that our design is beneficial for image-text matching and has good portability.