Reliable Benchmarking of Breast Ultrasound Lesion Classification Requires Patient-Level Validation

This study presents a leakage-aware benchmark of convolutional neural networks (CNNs), a Vision Transformer (ViT), and a CNN–transformer late-fusion configuration for benign-versus-malignant breast ultrasound classification on the BUS-UCLM dataset. After exclusion of normal-category images, the final cohort comprised 264 images from 36 patients, including 174 benign and 90 malignant images. Seven CNN-family models, one ViT baseline, and one ResNet18–ViT probability-level late-fusion configuration were evaluated using strict patient-level five-fold cross-validation. Additional analyses included fusion ablation, paired Wilcoxon signed-rank testing, Grad-CAM visualisation, and an image-level leakage demonstration. Under strict patient-level evaluation, performance was moderate across all models. GoogLeNet achieved the highest mean accuracy (61.48%), InceptionV3 achieved the highest mean Macro-F1 (59.24%), and ResNet50 achieved the highest mean AUC (0.6699), whereas the standalone ViT showed weaker overall discrimination. The late-fusion configuration remained competitive in threshold-dependent metrics but did not surpass the strongest CNN baselines in AUC. Overall, no architecture demonstrated a clear advantage across both threshold-dependent and threshold-independent metrics. By contrast, image-level splitting substantially inflated apparent performance, underscoring the importance of rigorous patient-level separation for credible benchmarking in breast ultrasound classification.