Multimodal deep learning with clinical metadata for skin lesion classification

Accurate skin lesion classification often requires integration of lesion appearance with patient clinical information. Although multimodal artificial intelligence (AI) approaches have shown promise, many existing methods rely on computationally intensive fusion mechanisms or simple integration strategies that may inadequately capture interactions between imaging and clinical data. This study aimed to develop and evaluate a multimodal deep learning model that integrates skin images and structured clinical metadata for skin lesion classification and to compare its performance with image-based and prior multimodal baseline models. This retrospective diagnostic study evaluated dataset-specific model performance using 2 public datasets (PAD-UFES-20 and ISIC 2019) and 1 institutional dataset (PKU-Skin). PAD-UFES-20 included 2,298 clinical images with 21 structured clinical variables across 6 lesion classes. ISIC 2019 included 25,331 dermoscopic images across 8 diagnostic categories. PKU-Skin was an institutional dataset of 800 clinical images (643 patients) with corresponding clinical metadata collected at Peking University Shenzhen Hospital and represented a binary classification task involving psoriasis and basal cell carcinoma. Model development and evaluation were performed separately within each dataset. The primary outcome was correct lesion classification. The primary evaluation metric was balanced accuracy. Secondary metrics included sensitivity, specificity, precision, F1 score, and overall accuracy. On PAD-UFES-20, the proposed multimodal model achieved a balanced accuracy of 85.6%, compared with 75.1% for a prior multimodal baseline model, with sensitivity of 76.1%, specificity of 95.2%, F1 score of 78.1%, and overall accuracy of 78.3%. On ISIC 2019, the model achieved a balanced accuracy of 86.0%, F1 score of 84.3%, and overall accuracy of 84.5%. On PKU-Skin, the model achieved a balanced accuracy of 99.6% and overall accuracy of 99.2%; because this dataset represented a separate binary classification task, these results should be interpreted as supportive evidence from an institutional setting rather than formal external validation of a single fixed model. In these retrospective dataset-specific experiments, integrating structured clinical metadata with skin images was associated with improved skin lesion classification performance relative to comparator models. These findings support further evaluation of multimodal approaches for dermatologic decision support, particularly in prospectively collected and clinically matched external datasets.