Design of an iterative hybrid multimodal deep learning method for early skin disease detection with cross-attention and graph-based fusions

This is the authors' abstract. We don't add key points for this paper.

This study proposes an end-to-end multimodal learning framework for early skin disease detection, incorporating both their spatial, temporal, and semantic information across heterogeneous patient data. The framework is composed of three key modules: (i)EfficientNet-B4 that extracts rich visual features from dermoscopic images, (ii) aBiLSTM enhanced with temporal attentionto model symptom evolvement from sensor-based time-series signals, and (iii) ClinicalBERT, a domain-specific transformer that generates contextual embeddings from patient clinical narratives. Modality-specific features are combined with a multi-head cross-attention mechanism to aggregate inter-dependency of input patterns and then fed into a Graph Attention Network (GAT) to capture inter-patient relationships according to feature affinity. This joint framework produces context-aware representations that can be used for classification. Experimental results show that the model can achieve predictive accuracy of 89.6 % and F1-score of 0.886 on average, which is superior to the state-of-the-art CNN-based baselines. Through simultaneously optimizing spatial detail, temporal dynamics, and clinical context, the Proposed SkinHarmoNet Model provides reliable and interpretable predictions, and its performance establishes the new state-of-the-art for multimodal dermatologic AI in a clinical setting.•Multimodal fusion: spatial, temporal, and semantic Modalities•Cross-attention and GAT: enhanced interaction of features•High performance: 89.6\ % accuracy, F1= 0.886.