FPGA‐Based Dual Path ConvNeXt‐Tiny–ViT With Effective Feature Refinement for Skin Cancer Classification

Skin cancer is the dangerous life‐threatening disease and is increasing rapidly due to environmental changes and lifestyle factors. The treatment for skin cancer is possible if detected at an early stage, but if not detected early, the possibility of successful treatment is reduced. Dermatologists use visual examinations to detect the type of skin cancer. However, this method is time‐consuming, requires high skill, and has limited accuracy, especially at early stages when melanoma and benign lesions look similar and are difficult to differentiate. In recent years, deep learning (DL) has gained popularity in image processing by producing excellent results. This research proposes a novel hybrid DL model, dual path ConvNeXt‐Tiny–vision transformer (ViT) with effective feature refinement. ConvNeXt‐Tiny extracts spatial and local features, while ViT extracts global contextual features. Feature refinement fuses all the features and selects the most important ones for classification. Two public datasets, ISIC‐2019 and HAM‐10000, are employed to evaluate the proposed network. The collected dataset is affected by class imbalance. To address this issue, a novel variational autoencoder generative adversarial network (VAE–GAN) network is introduced, which improves the quality of generated images and enhances classification accuracy. The proposed network is compared with other DL models, including AlexNet, DenseNet, and Transformer. The results show that the proposed network outperforms other DL models, achieving 99.38% accuracy on ISIC‐2019 and 99% on HAM‐10000. To assess the significance of the proposed system, an ablation study is conducted. The experimental outcomes of the ablation study highlight the effectiveness of the proposed network. Additionally, comparisons with state‐of‐the‐art (SOTA) methods are performed. After extensive evaluation, the proposed network is finalized and deployed on a field‐programmable gate array (FPGA) to serve as a low‐power, high‐speed edge device. A hardware accelerator is designed with high processing efficiency, optimized data transfer, dual cache, and efficient scheduling. The FPGA’s performance is evaluated using metrics such as power consumption, inference time, and throughput, and it is compared with GPU and CPU performance. The proposed accelerator classifies skin cancer 6.88× faster than a CPU and 2.29× faster than a GPU, while consuming only 2.8 W of power. The results demonstrate that the proposed FPGA‐based DL network is suitable for real‐time skin cancer classification, offering high speed, maximum accuracy, and low power consumption.