Infusing knowledge into CLIP via low-rank matrix for video-text retrieval

Video-text retrieval has been a crucial task with the exponential growth of video. Recent methods that leverage the pretrained CLIP model in the video-text retrieval task have demonstrated remarkable performance, surpassing many approaches trained on large-scale video-text datasets. However, these works ignore the performance-efficiency trade-off in the pursuit of better performance. Additionally, a completely new model is typically necessary for each task because many existing works fully fine-tune the pre-trained backbone. Therefore, to yield a compact and transferable model, we propose LoCLIP, a framework that transfers knowledge from CLIP in a parameter-efficient manner. Inspired by LoRA, we incorporate only a small set of trainable low-rank matrices per task, allowing adaptation to new tasks by simply replacing these matrices. In this way, we can acquire task-specific knowledge without compromising the prior knowledge stored in the pre-trained backbone. To demonstrate the effectiveness of our LoCLIP, we conduct extensive experiments and achieve comparable performance with state-of-the-art CLIP-based video-text retrieval methods while updating only a few parameters.