M2Former: Multiscale Patch Selection for Fine-Grained Visual Recognition

Recently, Vision Transformers (ViTs) have been actively applied to fine-grained visual recognition (FGVR). ViT can effectively model the interdependencies between patch-divided object regions through an inherent self-attention mechanism. In addition, patch selection is used with ViT to remove redundant patch information and highlight the most discriminative object patches. However, existing ViT-based FGVR models are limited to single-scale processing, and their fixed receptive fields hinder representational richness and exacerbate vulnerability to scale variability. Therefore, we propose MultiScale Patch Selection (MSPS) to improve the multiscale capabilities of existing ViT-based models. Specifically, MSPS selects salient patches of different scales at different stages of a MultiScale Vision Transformer (MS-ViT). In addition, we introduce Class Token Transfer (CTT) and MultiScale Cross-Attention (MSCA) to model cross-scale interactions between selected multiscale patches and fully reflect them in model decisions. Compared with previous Single-Scale Patch Selection (SSPS), our proposed MSPS encourages richer object representations based on feature hierarchy and consistently improves performance from small-sized to large-sized objects. As a result, we propose M2Former, which outperforms CNN-/ViT-based models on several widely used FGVR benchmarks.