MLTU: Mixup Long-Tail Unsupervised Zero-Shot Image Classification on Vision-Language Models

Abstract Vision-language models, such as Contrastive Language-Image Pretraining (CLIP), have demonstrated powerful capabilities in image classification under zero-shot settings. However, current Zero-Shot Learning (ZSL) relies on manually tagged samples of known classes through supervised learning, resulting in a waste of labor costs and limitations on foreseeable classes in real-world applications. To address these challenges, we propose the Mixup Long-Tail Unsupervised (MLTU) approach for open-world ZSL problems. The proposed approach employed a novel long-tail mixup loss that integrated class-based re-weighting assignments with a given mixup factor for each mixed visual embedding. To mitigate the adverse impact over time, we adopted a noisy learning strategy to filter out samples that generated incorrect labels. We reproduced the unsupervised results of existing state-of-the-art long-tail and noisy learning approaches. Experimental results demonstrate that MLTU achieves significant improvements in classification compared to these proven existing approaches on public datasets. Moreover, it serves as a plug-and-play solution for amending previous assignments and enhancing unsupervised performance. MLTU enables the automatic classification and correction of incorrect predictions caused by the projection bias of CLIP.