Robust and scalable representation learning for google street view–based neighborhood indicators

Understanding how visual representation models transfer across domains is critical for applying foundation models to real-world datasets. In this work, we present a systematic empirical study of representation learning for Google Street View (GSV) imagery, a domain that differs substantially from standard object-centric benchmarks such as ImageNet. We investigate how architecture choice, model scale, self-supervised post-training, and data curation influence downstream performance on neighborhood classification tasks. Specifically, we evaluate Vision Transformers (ViT), hierarchical Transformers (Swin), and state-space-based architectures (Vim and VMamba). To mitigate the domain gap, we initialize models with ImageNet-pretrained weights and further post-train them on one million unlabeled GSV images using DINO, a distillation-based self-supervised learning strategy. We then evaluate models with and without post-training under both full fine-tuning and linear probing protocols. Our experiments reveal several key findings. First, unsupervised post-training consistently improves full fine-tuning performance, especially when combined with target-aware clustering-based data curation. However, its effect under linear probing is task-dependent: post-training improves performance on Streetlight and Sidewalk but degrades performance on NSH and Green 30. This pattern is consistent with t-SNE visualizations, which show that these tasks form two distinct feature-aligned groups. Although target-aware sampling considers all downstream tasks when balancing the unlabeled post-training data, it does not fully remove this trade-off, likely because unsupervised post-training reshapes the representation space around dominant GSV visual patterns rather than the task-specific semantic cues required by each task. Full fine-tuning mitigates this drift, whereas linear probing remains limited by frozen features. Second, higher upstream ImageNet accuracy does not necessarily translate into stronger cross-domain transfer, and small-capacity models show performance saturation under extended post-training. Third, Mamba-based models are parameter-efficient and competitive but exhibit optimization instability during large-scale post-training, particularly at larger model scales. Finally, increasing the volume of uncurated post-training data yields diminishing returns, whereas curated data provides more consistent gains. Together, these findings show that effective cross-domain transfer depends jointly on architecture, scale, training strategy, and data curation, providing practical guidance for adapting vision foundation models to domain-shifted street-level imagery.