DermAssist: A Hybrid Vision Transformer System for Skin Lesion Diagnosis with Automated Alerting and Dual-Sided Portals

Abstract Skin cancer, one of the most prevalent forms of cancer globally, demands early and accurate diagnosis to improve patient outcomes. In this paper, we present DermAssist , a hybrid deep learning-based dermatology assistant system that integrates Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for automated multi-class skin lesion classification. Our model was trained and evaluated on widely-used datasets including DermNet, ISIC, and HAM10000, employing a robust preprocessing pipeline to enhance lesion visibility and diversity. DermAssist combines segmentation (U-Net), feature extraction (ResNet), and classification (BEiT Transformer), and introduces a dual-portal architecture for clinicians and patients using Streamlit and Flask interfaces. We incorporate Grad-CAM and SHAP for interpretability, Twilio-based SMS alerting for high-risk cases ( > 90% confidence), and secure AWS S3 storage to ensure HIPAA compliance. Experimental results demonstrate an accuracy of 90.2% with strong ROC-AUC and precision scores. DermAssist is positioned as a deployable, intelligent diagnostic aid capable of enhancing dermatological workflows in real-time environments.