MP47-13 ChatGPT EXPLAINS TESTOSTERONE THERAPY: ACCURATE ANSWERS WITH QUESTIONABLE REFERENCES

You have accessJournal of UrologySexual Function/Dysfunction: Medical, Hormonal & Non-surgical Therapy I (MP47)1 May 2024MP47-13 ChatGPT EXPLAINS TESTOSTERONE THERAPY: ACCURATE ANSWERS WITH QUESTIONABLE REFERENCES Jacob Hartman-Kenzler, Jose Torres, Arshia Alami-Harandi, Connelly Miller, Jiyeon Park, and William Berg Jacob Hartman-KenzlerJacob Hartman-Kenzler , Jose TorresJose Torres , Arshia Alami-HarandiArshia Alami-Harandi , Connelly MillerConnelly Miller , Jiyeon ParkJiyeon Park , and William BergWilliam Berg View All Author Informationhttps://doi.org/10.1097/01.JU.0001008880.11564.10.13AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Testosterone is a common focus of men's sexual health, for patients and clinicians. Given the sensitive nature of sexual health and recent broad availability of testosterone replacement therapy and supplements, patients may seek out anonymous sources of information. ChatGPT is an openly available artificial intelligence language learning model offering a conversation-based resource where patients may inquire about sexual health. This study aims to validate ChatGPT responses to questions about testosterone and evaluate the references used for its claims. METHODS: Nine questions about testosterone function and therapy were crafted based on American Urological Association (AUA) Care Foundation brochures for patient education on testosterone therapy and other commonly asked questions in men's sexual health clinics. ChatGPT was prompted to respond to these questions in a single interactive session with the phrase, "include references," to evaluate sources. Responses were separated into 53 statements backed by 21 references. Responses were evaluated by urology residents and faculty with AUA guidelines and core curriculum. References were evaluated on the accuracy of the authors, article, journal, and date of publication based on the PubMed index, Google Scholar, and the cited journal's website. RESULTS: The session yielded 53 statements backed by 21 references. Most queries yielded true statements by expert criteria (50/53, 94.3%). References were drawn from peer-reviewed journals (15/21, 71.4%); hospital websites (3/21, 14.3%), unknown sources (2/21, 9.5%) and AUA guidelines (1/21, 4.8%). The mean impact factor for peer reviewed journals was 27.2 (SD 60.5), and median 3.9. A majority of references were found to be legitimate (13/21, 61.9%). However, 4 (19.0%) included citation errors and 4 (19.0%) references were not found. The question yielding the most fabricated references (3/4, 75%) was "How do I boost my testosterone levels naturally?" CONCLUSIONS: ChatGPT is a broadly available conversation-based resource where patients may receive accurate information regarding general testosterone related questions. Our study demonstrates accurate responses from mostly accurate references in response to common questions. However, when queried about topics with fewer definitive references, the model fabricated sources with seemingly reputable authors and journals. This presents a serious pitfall of a natural language processing model for patients seeking reliable information; where false information can be presented backed up by illegitimate evidence. Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e768 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Jacob Hartman-Kenzler More articles by this author Jose Torres More articles by this author Arshia Alami-Harandi More articles by this author Connelly Miller More articles by this author Jiyeon Park More articles by this author William Berg More articles by this author Expand All Advertisement PDF downloadLoading ...