With the rapid development of multimodal big data modeling technology, cross-modal semantic alignment has become a major bottleneck limiting its performance improvement. Current research faces many problems such as high training costs, heterogeneous modal representation spaces, and a lack of fine-grained alignment. This paper studies a high-efficiency, high-precision cross-modal alignment method to overcome the semantic gap problem in traditional methods and improve the quality of multimodal data. Based on BLIP-2 (Bootstrapping Language-Image Pre-training), this paper constructs a two-level progressive alignment framework: First, using the Querying Transformer (Q-Former) as a learnable query transformer, a bidirectional Transformer framework is used to achieve cross-attentional interaction with the frozen visual encoder and self-attention modeling between queries, thus achieving preliminary visual-language alignment. In the second stage, the query representation output by Q-Former is projected onto the embedding space of the Large Language Model (LLM) through a fully connected layer, and deep adaptation between the visual representation and the generative language model is achieved through prefix hints. Experimental results show that the proposed method achieves a CIDEr of 138.2 on the COCO Caption image description task, a 3.7% improvement over BLIP, demonstrating excellent cross-domain transfer performance. It achieves an accuracy of 65.0% on the VQAv2 visual question answering task, realizing effective inference under zero-shot transfer. In the image-text retrieval task, the R@1 of text retrieval from Flickr30K images reaches 88.0%, validating the significant advantage in data efficiency. The conclusion proves that the two-stage freeze alignment strategy based on query-based interaction can achieve high-quality cross-modal semantic alignment with extremely low training costs, providing a feasible path for the lightweight deployment of multimodal generative AI.
Gao et al. (Thu,) studied this question.