What is the clinical evidence from this study?

Study design: Cross-Sectional. Population: Benign prostatic hyperplasia (BPH) (n=216). Intervention: ChatGPT-3.5. Primary outcome: Proportion of comprehensive and correct responses (Score 1).

September 30, 2025Scientific Reports1 citationsOpen Access

Assessment of the validity of ChatGPT-3.5 responses to patient-generated queries following BPH surgery

Q: What does this research mean for the field?

ChatGPT-3.5 provides comprehensive and correct responses to 78.2% of patient-generated queries regarding post-operative care following benign prostatic hyperplasia surgery. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

JNJad NajdiRoswell Park Comprehensive Cancer Center BABilal AlameddineAmerican University of Beirut Medical Center AAAlexandre ArmacheHofstra University

Key Result

ChatGPT-3.5 provided comprehensive and correct responses to 78.2% of patient-generated queries regarding post-operative care following benign prostatic hyperplasia surgery.

Study Design

Type

Cross-Sectional (n=216)

Structured PICO

Does ChatGPT-3.5 provide accurate and valid responses to patient-generated queries following BPH surgery?

Population

216 graded patient-generated queries regarding post-operative care following surgery for benign prostatic hyperplasia (BPH)

Intervention

ChatGPT-3.5 generated responses

Outcome

Validity and accuracy of responses assessed by urology residents using pre-defined criteria

ChatGPT-3.5 shows potential in answering post-operative BPH queries with 78.2% accuracy, though the risk of incomplete or misleading information necessitates further refinement.

Limitations

Used ChatGPT-3.5, a large language model not specifically trained on medical data
Lack of temporal consistency in responses due to continuous database updates
Included only U.S. based educational material and English speaking online material, limiting cross-cultural relevance
Did not involve patients or members of the public in the design, question selection, or evaluation phases
Only senior urology residents were involved in initial response evaluation
Potential reviewer bias and group-thinking during discrepancy resolution

Abstract

The rapid advancement of artificial intelligence, particularly large language models like ChatGPT-3.5, presents promising applications in healthcare. This study evaluates ChatGPT-3.5's validity in responding to post-operative patient inquiries following surgery for benign prostatic hyperplasia (BPH). Common patient-generated questions were sourced from discharge instructions, online forums, and social media, covering various BPH surgical modalities. ChatGPT-3.5 responses were assessed by two senior urology residents using pre-defined criteria, with discrepancies resolved by a third reviewer. A total of 496 responses were reviewed, with 280 excluded. Among the 216 graded responses, 78.2% were comprehensive and correct, 9.3% were incomplete or partially correct, 10.2% contained a mix of accurate and inaccurate information, and 2.3% were entirely incorrect. Newer procedures (Aquablation, Rezum, iTIND) had a higher percentage of correct answers compared to traditional techniques (TURP, simple prostatectomy). The most common errors involved missing context or incorrect details (36.6%). These findings suggest that ChatGPT-3.5 has potential in providing accurate post-operative guidance for BPH patients. However, concerns regarding incomplete and misleading responses highlight the need for further refinement to improve AI-generated medical advice and ensure patient safety. Future research should focus on enhancing AI reliability in clinical applications.

问 AI

Bookmark

View Full Paper