The classification of skin diseases from dermoscopy images is an important step in the early diagnosis of skin diseases and in providing clinicians with decision support. While existing deep learning techniques use only visual data to make diagnostic decisions, they do not take into account other clinically relevant information such as a patient’s gender, age, or location of the lesion. Because of the limitations of these unimodal systems, the robustness of a diagnostic decision can be severely compromised when attempting to diagnose skin diseases that are both visually ambiguous and underrepresented. DermaFusionNet is proposed in this paper as a novel multimodal deep learning architecture that combines dermoscopic image features with structured clinical metadata using a gate-based cross-modal fusion mechanism. The architecture described in this paper incorporates ConvNeXt visual feature extractors and dedicated metadata encoders, and it also employs an adaptive gated module to adjust the effect of each modality to ensure stable and contextually relevant feature integration across modalities. To further enhance the stability of multimodal training, the authors have included auxiliary modals to reduce the potential for one modality to dominate during training. The authors demonstrate the effectiveness of DermaFusionNet by evaluating it using the HAM10000 dataset and a stratified experimental protocol. The experimental results show that DermaFusionNet achieves an overall classification accuracy of 89.22%, with a precision of 86.32%, a recall of 89.22%, and an F1-score of 89.08%. Furthermore, DermaFusionNet outperformed not only the image-only baseline but also the naïve fusion baseline. The authors conclude that the use of gated multimodal fusion techniques has the potential to improve the reliability of automated skin disease classification systems.
Kanani et al. (Wed,) studied this question.