Evaluating four consumer artificial intelligence chatbots on smartphone dermatology images: diagnostic accuracy, the contribution of clinical context, and skin-tone equity

Patients increasingly upload smartphone photographs to consumer artificial intelligence (AI) chatbots seeking dermatological diagnoses, yet accuracy on such images is unexplored. We compared the diagnostic accuracy and equity of four leading AI chatbots across varying input conditions. GPT-5.4-Pro, Claude-Opus-4.6, Gemini-3.1-Pro and Grok-4.20-Reasoning were evaluated on 3,005 cases from the crowdsourced Skin Condition Image Network (SCIN) dataset with physician-validated ground truth. Each case was assessed under four conditions: single image (n = 1,073), multiple images (n = 1,932), images plus clinical context (n = 2,567) and text-only (n = 2,567), using top-1 and top-3 hierarchical matching. In the primary analysis (images plus context), models clustered into two tiers: Grok (53.2%, 95% CI 51.3–55.1) and Gemini (52.6%, 95% CI 50.7–54.5) significantly outperformed GPT (45.5%, 95% CI 43.5–47.5) and Claude (44.6%, 95% CI 42.7–46.5) for top-1 accuracy (Cochran’s Q, p < 0.001). Gemini led top-3 accuracy at 79.3% (95% CI 77.7–80.8). Adding clinical context improved accuracy for Grok (+ 5.5%, p < 0.001) and GPT (+ 3.8%, p = 0.003) only, whereas removing images caused losses of 11 to 18 percentage points across all models. Inter-model agreement was low across image-based analyses (Fleiss' kappa 0.34–0.41). For top-1 accuracy, Fitzpatrick skin tone disparity was significant only for Grok, the best-performing model (56.0% light versus 45.1% dark skin, p = 0.007), whereas top-3 disparities were significant for Claude, Gemini and Grok. Consumer AI chatbots show promise, but skin tone disparities, low inter-model agreement and content moderation limitations indicate these tools are not ready for standalone clinical use.