Answers are grounded in this paper and cite their sources.
We address the task of weakly-supervised few-shot image classification and segmentation, by leveraging a Vision Transformer (ViT) pretrained with self-supervision. Our proposed method takes token representations from the self-supervised ViT and leverages their correlations, via selfattention, to produce classification and segmentation predictions through separate task heads. Our model is able to effectively learn to perform classification and segmentation in the absence of pixel-level labels during training, using only image-level labels. To do this it uses attention maps, created from tokens generated by the self-supervised ViT backbone, as pixel-level pseudo-labels. We also explore a practical setup with “mixed” supervision, where a small number of training images contains ground-truth pixel-level labels and the remaining images have only image-level labels. For this mixed setup, we propose to improve the pseudo-labels using a pseudo-label enhancer that was trained using the available ground-truth pixel-level labels. Experiments on Pascal-5 i and COCO-20 i demonstrate significant performance gains in a variety of supervision settings, and in particular when little-to-no pixel-level labels are available.
No takes yet. Share an insight, caveat, or question.
Kang et al. (2023) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: