Evaluating large language models for skin lesion diagnosis and triage in rural settings

Introduction Skin cancer is a growing global cause of morbidity and mortality. Rural populations are at an increased risk due to greater ultraviolet exposure and limited access to dermatology services. Although large language models (LLMs) are increasingly used for health information, their ability to accurately diagnose and triage skin lesions remains uncertain. Objectives To evaluate the diagnostic accuracy, triage safety and reproducibility of multimodal LLMs in interpreting skin lesions representative of rural clinical practice. Methods Seven multimodal LLMs from three developers (OpenAI, Google, and Anthropic) were evaluated using 34 histopathologically confirmed skin lesions from a rural Canadian practice. The dataset comprised paired macroscopic and dermoscopic images of both malignant and non-malignant lesions. Each image was assessed independently a total of five times by each model using a standardized prompt. Results A total of 2,380 model outputs were generated. Diagnostic accuracy varied widely across models (39.4%-72.1%). Claude Opus 4.8 achieved the highest accuracy (72.1%) and the best overall balance of sensitivity, specificity, and reproducibility. GPT-5.5 Instant had the lowest specificity and highest over-triage rate, whereas Gemini 3.5 Flash had the lowest sensitivity and highest misclassification rate. Safe triage of malignant lesions ranged from 36% to 87%. Dermoscopic images significantly improved AI performance compared with macroscopic images (OR, 1.65; 95% Cl, 1.30–2.10; p < 0.001). Self-reported diagnostic confidence correlated poorly with model performance, and diagnostic reproducibility across repeated runs was moderate to almost perfect for most models. Conclusions Multimodal LLMs can interpret skin lesion images but are not yet accurate, consistent, or safe for independent diagnosis or triage. Higher-performing models may have a role as supervised decision-support tools, but autonomous use risks diagnostic errors and could worsen rural healthcare inequities. Clinical trial number Not applicable.