Systematic Failure of Vision Transformers in Imbalanced Skin Lesion Classification

Background/Objectives: Vision Transformers (ViTs) have demonstrated impressive performance in dealing with large-scale natural image datasets. They have started to be used in medical image classification problems as well. However, how they behave under real-world conditions, such as data scarcity and extreme class imbalance, has not been well investigated. In this study, we examine the feasibility of using a standard Vision Transformer Base model that learned from scratch how to classify skin lesion images into multiple classes using the ISIC 2019 dataset. Methods: The Vision Transformer architecture was trained from scratch using stratified splitting of the data, class-balanced cross-entropy loss, multi-seed initialization, and control of hyperparameters such as patch size and dropout rate. The evaluation of the Vision Transformer architecture was performed using a hold-out test set with metrics such as accuracy, macro-F1, weighted-F1, and analysis of the confusion matrix. Results: Across all configurations, the training exhibited substantial instability and consistent overfitting behavior, with an average accuracy gap between validation and test sets of 22.7%. Test accuracy ranged from 8.0% to 37.8%, showing high sensitivity to initialization. For minority classes, the F1-score remained very low (F1 < 0.05) even though the classes were balanced in the loss function. Conclusions: The results indicate that a standard ViT-Base model trained from scratch can exhibit pronounced instability and a tendency toward majority-class bias when applied to multi-class skin lesion classification under conditions of extreme class imbalance and data scarcity. The findings point to the limitations of using simple transformer models without pre-training or other forms of inductive bias in scarce data settings.