(874) The Performance of Artificial Intelligence Models in Providing Post-Operative Care Recommendations for Adrological Surgeries

Introduction Patients have begun using artificial intelligence (AI) models more frequently in recent years. Likewise, researchers have also been focused on AI models and tested their potential in various concepts. The knowledge base of AI models regarding diseases and their ability to assist patients with disease and treatment processes has been extensively studied. However, their ability to provide disease-specific recommendations for post-operative recovery has not been thoroughly investigated. Objective We aimed to assess the performance of AI models in providing post-operative care recommendations after adrological surgeries Methods Case scenarios for 6 different andrological surgeries (varicocele, penile prosthesis, testicular sperm extraction, penile plication, penile fracture, and shunt surgery for priapism) were asked to ChatGPT, Copilot, Gemini, DeepSeek in the format of “I had …….. surgery, provide post-operative care instructions for me.” and the outputs were recorded. The Gunning Fog Index and Simplified Measure of Gobbledygook (SMOG) were used for readability assessment. The understandability and actionability of the answers were evaluated by the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P). Additionally, the presence of reference, reference source, and satisfaction with the output were assessed by a 5-point Likert scale (1 very dissatisfied, 2 dissatisfied, 3 neither dissatisfied or satisfied, 4 satisfied, 5 very satisfied) Results The readibility indexes and PEMAT-P scores of AI models were listed in Table. There was no statistically significant difference between ChatGPT, Copilot, Gemini, and DeepSeek regarding readability indexes and PEMAT-P understandability and actionability scores. The mean Gunning Fog score was 10.6±0.9 and the mean SMOG score was 8.8±0.8. The mean PEMAT-P understandability and actionability scores of AI models were 81.4±5.9 and 62.5±6.7 respectively. The satisfaction with ChatGPT output was reported as very satisfied in 3/6 (50%), neither dissatisfied or satisfied in 2/6 (33.3%), and satisfied in 1/6 (16.7%). The satisfaction with Copilot output was reported as satisfied in 4/6 (66.7%) and neither dissatisfied or satisfied in 2/6 (33.3%). The satisfaction with Gemini output was reported as very satisfied in 4/6 (66.7%) and satisfied in 2/6 (33.3%). All outputs of DeepSeek were classified as very satisfied (6/6, 100%). ChatGPT cited every output it produced, whereas Copilot and DeepSeek did not present references in any of their outputs. Gemini listed reference for only 1/6 (16.7%) output. Gemini’s only reference source was hospital website. ChatGPT’s reference sources were hospital websites in 6 outputs, medical associations’ websites in 3 outputs, medical articles in 3 outputs, and private doctors’ websites in 3 outputs. Conclusions In terms of providing post-operative care recommendations for andrological surgeries, ChatGPT, Copilot, Gemini, and DeepSeek produce equally readable, understandable, and relatively actionable outputs. However, the quality of the output and number of outputs with reference vary significantly among AI models. Disclosure No