(741) Evaluating AI Tools for Scrotal Pain: Comparing Clinical Guidance and Safety

Introduction Artificial intelligence (AI) chatbots have become increasingly common tools for patients seeking quick information about urologic symptoms, including scrotal pain. Since these conditions can range from benign causes to surgical emergencies, ensuring that chatbot responses are accurate, safe, and empathetic is essential to prevent misinformation and delayed care. Objective This study aimed to evaluate whether ChatGPT and Grok can serve as reliable tools for the initial triage of scrotal pain by assessing their ability to identify potential causes, provide appropriate educational information on treatment options, and recognize cases that require urgent medical evaluation. Methods Sixty-one standardized questions related to scrotal pain were submitted to ChatGPT (version 5 omni [ChatGPT-5o], OpenAI) and Grok (free web version, xAI). Questions were categorized into five domains: testicular torsion, epididymitis/orchitis, varicocele, chronic idiopathic orchialgia, and referred pain. Two urologists independently rated all responses. Information quality was assessed using the DISCERN instrument (1–5 scale), while understandability and actionability were measured using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P; 0–100%). Misinformation was rated from 1 (no misinformation) to 5 (high misinformation), and safety from 0 to 2. Statistical and graphical analyses were performed using RStudio (version 2023.12.1+402). Results Across 61 questions assessed, mean DISCERN scores were significantly higher for Grok than ChatGPT (4.29 vs 3.81, p < 0.001). PEMAT Actionability scores were also higher for Grok (85.5% vs 79.9%, p = 0.02), whereas PEMAT Understandability scores were comparable between models (86.3% vs 86.4%, p = 0.14). Misinformation scores were similar between chatbots (2.04 vs 2.09, p = 0.11). Safety ratings were significantly higher for Grok (1.68 vs 1.52, p = 0.046). Conclusions Both ChatGPT and Grok provided generally reliable information for the initial triage of scrotal pain. Grok demonstrated superior information quality, greater actionability, and higher safety ratings, while both models achieved comparable understandability and low misinformation levels. These findings highlight the potential of AI chatbots to support patient education on diagnosis and treatment options, though professional evaluation remains indispensable for accurate diagnosis and management. Disclosure No