SkinFormer: a hybrid vision transformer and ConvNeXtV2 approach for skin cancer detection and segmentation

Skin cancer remains one of the most prevalent and deadly forms of cancer globally, where early detection plays a critical role in improving survival rates. In this study, we introduce SkinFormer, a novel hybrid deep learning architecture primarily designed for binary skin lesion classification (benign vs. malignant), enhanced by a standalone pretrained segmentation module for accurate ROI cropping. It incorporates Vision Transformers (ViT), ConvNeXtV2, and a Separable Self-Attention method to improve the precision and interpretability of skin lesion classification and segmentation. Utilizing a portion of the ISIC 2024 dataset alongside the DermQuest dataset, which includes more than 55,000 images for binary classification of benign and malignant skin lesions, SkinFormer attains promising performance with an accuracy of 96.8%, an AUC of 0.985, and an F1-score of 95.8%. The independent segmentation module attains a Dice Similarity Coefficient of 95.6% and an Intersection over Union of 91.8%. The approach effectively resolves issues including irregular lesion borders, low contrast, and artifacts through the integration of global contextual reasoning and local texture extraction. Ablation experiments validate the importance of each architectural module, while qualitative heatmaps and overlay visualizations enhance its clinical interpretability. Statistical significance was established by McNemar’s test (p < 0.01), underscoring the method’s resilience. The results indicate that SkinFormer is a potential instrument for automated skin cancer diagnosis, a potential tool for clinical decision support and teledermatology applications.