BACKGROUND Artificial intelligence (AI) models are being increasingly integrated into clinical care. Moreover, the accessibility of open-AI resources makes them attractive to patients seeking clinical information. Little is known regarding the utility of large language models as patient resources for navigating major cancer diagnoses. OBJECTIVE This study evaluated the content, readability and safety of ChatGPT 4.o-generated responses to common perioperative queries about hepatic, pancreatic, and colon cancers. METHODS A 28-question survey was developed based on frequently asked surgical questions for select malignancies. Surgical oncologists rated ChatGPT 4.o-generated responses on a 5-point Likert Scale for accuracy, quality, and tangibility. Readability was assessed using the Flesch-Kincaid Reading Grade Level (FKRGL). Respondents provided free-text comments and reported comfort with patients using ChatGPT. Survey completion implied consent. RESULTS Seven attending surgical oncologists with a median of 7 years in practice completed the survey. Responses received mean scores of 3.5/5 for quality, 3.6/5 for accuracy, and 3.6/5 for tangibility. The responses had a median FKRGL score of 14.5. On post-hoc analysis for select questions, median FKRGL was 15.6, decreasing to 7.1 and 14.5 with prompting and rephrasing. Numerous inaccuracies and content gaps were reported, and approximately 43% of providers did not report feeling “comfortable” in having patients consult publicly available AI for medical information. CONCLUSIONS This study provides cautionary, yet optimistic, findings regarding the value of open-source ChatGPT as a patient resource for abdominal malignancies. Providers should be prepared to effectively counsel patients concerning use of ChatGPT to mitigate minor inaccuracies and address readability challenges.
Lindsay et al. (Sun,) studied this question.