A patient‐oriented, general‐practitioner‐level, deep‐learning‐based cutaneous pigmented lesion risk classifier on a smartphone

Dear Editor, Given that advanced melanoma has a poor prognosis, early detection is essential.1 Recently, deep learning (DL) models have shown promise in aiding diagnosis of melanoma.2,3 However, these models consider only images, and not complementary clinical information, and are mainly for diagnostic purposes rather than for screening.4 These may limit potential clinical benefits from the DL technology. Herein we report patient‐oriented, DL‐based cutaneous pigmented lesion screening models that take smartphone‐generated images and clinical information simultaneously. This study was approved by the institutional review board of Taipei Medical University (TMU‐JIRB: N201711077). We collected data from 4635 Taiwanese participants who gave informed consent. Questions designed to obtain clinical information were as follows (answers; distribution): Q1. Sex (male, female, not disclosed; 56%, 43%, 1%); Q2. Age group (≤ 20, 21–40, 41–65, ≥ 66 years; 12%, 61%, 26%, 1%); Q3. Mole size > 0·6 cm? (yes, no; 3%, 97%); Q4. How long has the mole existed? (≤ 1 month, 1 month to 1 year, ≥ 1 year, uncertain; 2%, 11%, 85%, 2%); Q5. Has the mole changed in the last month, for example in size, colour, shape, bleeding or itching? (yes, no, uncertain; 6%, 90%, 4%). Participants were instructed to photograph their relevant moles using their own smartphone cameras (≥ 6 megapixels required). For data generalization, we asked 1736 participants to photograph a lesion twice from different distances (5 and 15 cm). A total of 6371 images were collected and reviewed by three board‐certified dermatologists (Y.C.J.L, H.H.W and Y.T.L, with 22, 10 and 8 years of clinical experience). They first excluded 1082 images with insufficient quality and annotated the remainder (image plus clinical information) as ‘higher risk’ or ‘lower risk’ based on consensus. This resulted in 937 higher‐risk and 4352 lower‐risk cases in the dataset. Unanimous consensus was reached in 99% of cases, which can be explained by the fact that judging whether a pigmented lesion is at risk is straightforward for a professionally trained dermatologist based on patient age and the ‘ABCDE’ rule. For model construction, we first performed stratified sampling to create a test set of 120 cases for later comparison of performance between models and general practitioners (GPs). We then performed stratified sampling with 20% of the remaining dataset to create a validation set, while the remainder served as a training set. For data balancing, augmentation was performed as previously described,5 generating a final training set of 8000 lower‐risk and 8000 higher‐risk cases. Areas under the receiver operating characteristic (ROC) curves (AUCs) of trained models were calculated from the validation set. Comparisons of ROC curves were performed as described previously.6 The optimal threshold was determined by Youden index. We first examined the effect of combining image and nonimage clinical information on DL models. Three frameworks, including image (Framework_I), clinical information (Framework_C) and the combination (Framework_I+C), were designed for three convolutional neural network architectures (DenseNet121, VGG16 and ResNet50). For Framework_I, AUCs were 0·84 ± 0·04 (sensitivity 0·74 ± 0·06, specificity 0·86 ± 0·02), 0·83 ± 0·03 (sensitivity 0·72 ± 0·07, specificity 0·86 ± 0·02) and 0·86 ± 0·03 (sensitivity 0·77 ± 0·06, specificity 0·85 ± 0·02) for DenseNet121, VGG16 and ResNet50, respectively. Pairwise comparison of AUCs found no statistical difference (P > 0·05). In contrast, Framework_I+C achieved AUCs of 0·92 ± 0·003 (sensitivity 0·93 ± 0·04, specificity 0·86 ± 0·02), 0·93 ± 0·02 (sensitivity 0·95 ± 0·03, specificity 0·87 ± 0·02) and 0·89 ± 0·02 (sensitivity 0·89 ± 0·05, specificity 0·77 ± 0·03) for DenseNet121, VGG16 and ResNet50, significantly outperforming their Framework_I and Framework_C counterparts (P < 0·05 for all comparisons). Ensemble of the three Framework_I+C models further increased the AUC to 0·94 ± 0·02 (sensitivity 0·96 ± 0·03, specificity 0·87 ± 0·02), which was overall higher than that of each model alone (P < 0·05, P = 0·064 and P < 0·05 vs. DenseNet121, VGG16 and ResNet50, respectively). We then compared the ability for risk classification between our best‐performing ensemble model (I+C) and 23 human GPs (median clinical experience 10 years). The model demonstrated superior performance to the majority (22 of 23) of the GPs, whose sensitivity and specificity points lay below the model's ROC curve (Fig. 1). The ensemble deep learning model achieved human general practitioner (GP) level in classification of cutaneous pigmented lesion risk. The comparison of performance in the classification between the ensemble deep learning model (Ensemble_I+C) and 23 human GP evaluators was made using the receiver operating characteristic (ROC) curve of the model (red) and the sensitivity–specificity distribution of the GPs. The green triangles represent the sensitivity and specificity of individual GPs, with the average denoted by the blue cross. The results show better performance of the model compared with the majority of the GPs, whose sensitivity and specificity points lie below the ROC curve of the model. To our knowledge, this is the first patient‐oriented, DL‐based, GP‐level pigmented lesion screening model that can process both images and nonimage clinical information, making it more clinically pertinent than other currently available DL models.7,8 Also, our models are smartphone compatible as they are trained on patients’ smartphone‐generated images. Our concept may be extrapolated to other dermatological conditions such as psoriasis and acne, which are typically evaluated based on both clinical information and lesion appearance. There are limitations in this study. Firstly, our participants are all ethnically Asian and mostly under the age of 40 years, a population generally considered at low risk for melanoma. Future work will incorporate data from different ethnicities and age groups. Secondly, for data annotation, instead of using histological examination, we adopted clinical evaluation results from experienced, board‐certificated dermatologists mainly because it is not justified to biopsy ‘lower‐risk’ lesions. We are constantly contacting the ‘higher‐risk’ patients to obtain their biopsies. A follow‐up study is under way to determine the clinical impact of our DL model. We would like to acknowledge the contributions of Dr Yu‐Chen Huang and Dr Yin‐Yi Chiang for collecting clinical data. Funding sources: This work was partially funded by the Ministry of Science and Technology, Taiwan. Conflicts of interest: Y.C.J.L. and Y.P.H.C. are founders of DermAI Co. Ltd.