Multimodal models for skin cancer classification using clinical freetext and dermatoscopic images

Skin cancer is one of the most prevalent cancers globally, with early detection critical to ensure reduced mortality risk. To aid early detection, machine learning (ML) skin cancer detection models have been proposed, currently with a focus on dermatoscopic imaging only. However, freetext may provide extra diagnostic information that is not present in images alone. We constructed a multimodal dataset comprising 5481 dermatoscopic images from 4538 patients, including patient metadata and clinical notes, with binary labels (benign vs. malignant, 7% malignant). To assess and mitigate bias from leading language, we developed a clinical text preprocessing pipeline combining regular expressions and large language models, enabling multiple levels of filtering. We train multimodal ML models on this dataset to explore the effect of freetext on model performance. Our results show that incorporating unfiltered text significantly improves classification performance (0.970 AUROC) compared to visual data alone (0.909 AUROC); even with leading language removed, performance gains persist (0.948 AUROC). This work benchmarks clinical freetext inclusion in skin lesion classification, demonstrating that clinical text contributes predictive value beyond that available in images alone. The model’s high performance on unfiltered clinical text highlights the high levels of bias, and possible shortcutting, present in this text which may make it unsuitable for inclusion in some ML models. By systematically filtering clinical notes via our proposed technique, we show that multimodal models retain improved accuracy while reducing bias. These results provide practical guidance for integrating clinical text into real-world skin cancer detection systems and establish a foundation for future multimodal research in dermatology. Prompt detection of skin cancer improves survival, but diagnosis must be made by clinicians. Image-based machine learning models for skin cancer classification have shown promise. However, key information is often only recorded in clinical notes, such as whether a lesion has changed, itches, or bleeds. By creating a dataset that contains images, patient data, and freetext descriptions of the problem, we train a series of machine learning models on both images and freetext to predict skin cancer. We show that the inclusion of freetext significantly enhances model performance, but that care must be taken to ensure the freetext does not unintentionally bias the model. These models could be used in multiple points in a skin cancer clinical workflow to either support more accurate referrals to dermatology, or direct patient access to dermatology services, potentially reducing wait times and improving patient outcomes. Watson et al. explore multimodal machine learning models for lesion classification, using dermatoscopic images, freetext, and patient metadata; they investigate how leading language in freetext affects model bias, and introduce methods to address this. Their results show freetext improves model performance even with the leading language removed.