DINOv3–SAM: Fusing Semantic and Anatomical Foundation Models for Robust Medical Image Classification

Robust medical image classification remains a persistent challenge due to heterogeneous acquisition protocols, label noise, and subtle inter-class variability. In chest radiography, these difficulties are compounded by shortcut learning from non-anatomical artifacts and projection variability, which severely undermine generalization. To address these limitations, we propose DINOv3–SAM, a dual-encoder framework that synergistically integrates complementary visual representations. The architecture combines a self-supervised DINOv3 vision transformer, which captures global semantic context, with the Segment Anything Model (SAM) image encoder, which provides anatomy-aware spatial features. The two feature streams are spatially aligned and adaptively refined through a Convolutional Block Attention Module (CBAM), which emphasizes informative channels and diagnostically relevant regions prior to classification. Notably, only the final DINOv3 layers are fine-tuned while SAM remains frozen, yielding a computationally efficient framework with only 5.3 million trainable parameters. We evaluate DINOv3–SAM across five public datasets spanning diverse modalities and diagnostic tasks: pediatric pneumonia classification, cardiomegaly detection, large-scale RSNA pneumonia detection, COVID-19 radiography, and dermatological lesion classification. A unified stratified train/validation/test protocol ensures consistent and fair evaluation across both binary and multi-class settings. Each configuration was trained three times with different random seeds, and comprehensive comparisons against reproduced CNN baselines and published state-of-the-art methods, supported by statistical significance testing, demonstrate improvements across most datasets and metrics. The proposed framework achieves a mean accuracy of 94.94% (±3.06) on pediatric pneumonia, 97.24% (±0.77) accuracy (AUC 0.998 ±0.001) on COVID-19 radiography, and a mean AUC of 0.8705 (±0.0445) on the challenging RSNA benchmark. These results underscore the efficacy of fusing self-supervised semantic representations with segmentation-informed anatomical cues to enhance robustness against dataset bias, acquisition variability, and cross-domain shifts. The proposed approach offers a reliable foundation for multi-dataset and multi-modality medical image classification in research-oriented decision-support systems.