Abstract The template is paramount for visual object tracking, which is treated as an object model to recognize the tracked object in the following images. Thanks to Segment Anything Model 2 for providing the powerful capability to perform feature extraction and fusion on multi-frame images, achieving accurate object tracking. However, since it uses consecutive adjacent frames as templates, the problem of error accumulation is more likely to occur during the tracking process, which limits its long-term tracking performance. Therefore, we first demonstrate through extensive experiments that the tracking performance can be significantly improved by using a naive template filtering mechanism. Subsequently, inspired by this experimental result, based on the principle of reducing cumulative error, a multi-level memory screening structure is designed to form a dynamic template set. In the LaSOT Test dataset, without any model fine-tuning, our method ultimately achieves AUC scores of 74.7%, 72.63%, 72.18%, and 71.72% with the large, base plus, small and tiny model, respectively, surpassing the SAMURAI method and approaching the performance of DAM4SAM. Meanwhile, we also evaluated performance on the more challenging LaSOText dataset, where MAS4SAM outperforms the original SAM2 by 4.61%, 3.8%, 3.07%, and 4.59% across the four backbone scales, respectively. Furthermore, it yields more significant improvements than the fixed threshold method in both subsets of LaSOT, indicating that MAS4SAM exhibits adaptive behavior in template screening, thus improving the long-term tracking performance of SAM2 in a wide range of scenarios. The code is released at https://github.com/Cannol/MAS4SAM.
Li et al. (Mon,) studied this question.