Beyond Accuracy: Methodological Evaluation of Performance Stability in Deep Learning Models on the HAM10000 Multi-Class Skin Lesion Benchmark

Objective: Deep learning models are widely used in medical imaging and often achieve high accuracy; however, evaluations based on a single metric may fail to capture performance variability in imbalanced multi-class problems. This study quantitatively investigates the sensitivity of deep learning models to data splitting and random initialization within a single benchmark dataset for multi-class classification of dermoscopic skin lesions. Material and Methods: A 3-fold × 3-seed iterative cross-validation framework was applied to the HAM10000 dataset using EfficientNetB0 and MobileNetV2 architectures. Model performance was assessed using accuracy, macroaveraged precision, recall, F1-score, and multiclass AUC. For each metric, mean and standard deviation values were calculated across folds and random seeds. Results: The MobileNetV2 model achieved an average accuracy of 0.725±0.025, while the EfficientNetB0 model achieved an average accuracy of 0.715±0.045. In contrast, the macroaverage F1 scores were calculated as 0.461±0.155 for EfficientNetB0 and 0.531±0.056 for MobileNetV2, with particularly high variability observed in the EfficientNetB0 model. Similarly, while accuracy values reached 74% in some layer and seed combinations, the F1-macro values dropped to the range of 0.27-0.41. Variance decomposition revealed that variability in macro-averaged metrics was predominantly driven by cross-validation fold composition, whereas random initialization contributed a comparatively smaller proportion of total variance. Conclusion: The findings obtained according to the HAM10000 dataset reveal that although the accuracy values of the models show relatively stable performance, class-balanced performance metrics can show significant fluctuations.