How SAM helps Unsupervised Video Object Segmentation?

This is the authors' abstract. We don't add key points for this paper.

As an emerging vision foundation model, Segment Anything Model (SAM) has been successfully applied to Video Object Segmentation (VOS). However, previous methods rely on first frame’s mask or manual interaction, which belong to semi-supervised or interactive video object segmentation. The potential of SAM in Unsupervised Video Object Segmentation (UVOS) remains to be explored. In this work, we propose a two-stage training framework to explore how SAM helps UVOS. In Stage 1, utilizing SAM powerful feature extraction ability, we only train a lightweight decoder with feature aggregation module. This stage employs relaxed flow reconstruction as an unsupervised proxy task for object discovery. In Stage 2, based on SAM promptability, we design a refinement process that involves rough correction based on in-clip temporal consistency and heuristic refinement guided by Prompt-closure, to generate refined results from the coarse mask produced by Stage 1. Finally, update the segmentation head using refined results along with corresponding confidence. The proposed method demonstrates competitive performance on three common UVOS datasets (DAVIS2016, SegTrackv2, FBMS59) with higher accuracy, faster convergence, lower training cost, validating the effectiveness of our approach.