Leakage-Aware and Reproducible Evaluation of Deep Learning Models for Multi-Class Oral Disease Image Classification

Duplicate and near-duplicate images can compromise deep learning evaluation through cross-partition leakage. This study investigated duplicate-aware evaluation and reproducibility using 667 public oral photographs representing Caries (219), Mouth Ulcer (265), and Tooth Discoloration (183). Exact and near-duplicate content was examined using cryptographic and perceptual hashing, supplemented by human-reference assessment of 378 image pairs. The automated audit initially identified 580 duplicate-content groups, refined to 564 following human validation. A corrected group-disjoint split (466 training, 101 validation, 100 test images) was used to evaluate ImageNet-pretrained ResNet50, EfficientNet-B3, and ViT-B/16 across three seeds. Performance, calibration, and uncertainty were assessed using a standardized protocol and 10,000 paired duplicate-group bootstrap resamples. ViT-B/16 achieved the highest mean accuracy (92.67% ± 1.15 percentage points) and macro-F1 (0.9194 ± 0.0138), followed by ResNet50 (92.00% ± 2.00; 0.9135 ± 0.0213) and EfficientNet-B3 (83.33% ± 2.08; 0.8263 ± 0.0224). The ViT-B/16–ResNet50 accuracy difference was 0.67 percentage points (95% confidence interval: −3.33 to 4.67), indicating uncertain relative superiority. Exploratory evaluation on 1786 externally annotated caries-positive images yielded mean sensitivities of 87.38%, 74.25%, and 69.65% for EfficientNet-B3, ResNet50, and ViT-B/16, respectively, with substantial seed variability. These findings support human-validated duplicate auditing, group-disjoint partitioning, multi-seed testing, and paired uncertainty analysis. Limited disease coverage, unavailable patient identifiers, and positive-only external evaluation constrain conclusions regarding multiclass generalizability and clinical applicability.