Generative artificial intelligence (AI) models have become increasingly accessible and advanced with multimodal input. Infantile hemangiomas (IHs) are the most common pediatric vascular tumor, but pediatricians may have difficulty distinguishing them from similar-appearing lesions. We assess the capability of a public AI model, ChatGPT 4.0 (GPT) to distinguish between IHs and other vascular anomalies (VAs), evaluating its potential role as a clinical tool for pediatric clinicians. This retrospective study assessed 50 IH and 50 non-IH VA images using a GPT zero-shot approach with the binary task of diagnosing an IH (or not). The same images were provided to four general pediatricians; comparison was performed between pediatricians and GPT. GPT achieves 75% accuracy in IH identification with an F1 score of 0.742. ROC curve generation yields an AUC of 0.80. Hundred-image analysis demonstrates that actual diagnosis affects GPT accuracy (p = 0.015). 50-IH-image analysis reveals configuration (p = 0.027), skin phototype (p = 0.019), and anatomical location (p = 0.047) as factors that may affect GPT accuracy. Comparing GPT to pediatricians reveals comparable results (p = 0.345). This off-the-shelf publicly available GPT (75%) was less accurate than a previously published well-trained AI that achieved higher accuracy (~92%). GPT's F1 score of 0.742 indicates moderate balance between precision and sensitivity, and an AUC calculation of 0.80 indicates a potential role for GPT as a clinical assistant in primary care settings. An untuned commercial GPT does not yet outperform pediatricians despite studies of well-trained AI. This study provides variables impacting GPT accuracy, serving as a foundation for future research.
On et al. (Thu,) studied this question.