The rise of ChatGPT-4: exploring its efficacy as a decision support tool in esophageal surgery – a research letter

Q: What is the clinical evidence from this study?

Study design: Letter. Population: Esophageal disorders (esophageal cancer, Barrett's esophagus, gastroesophageal reflux disease, achalasia of cardia) (n=35). Intervention: ChatGPT-4 (Version: Nov 2023) vs. ISDE and NCCN clinical guidelines evaluated by three esophageal surgery specialists. Primary outcome: Commendable clinical response rate (median expert score of 2 or 3 on a 0-3 scale).

Q: What does this research mean for the field?

ChatGPT-4 provides commendable, guideline-aligned responses to 97.2% of clinical inquiries in esophageal surgery, with 68.5% achieving the maximum median expert score. Novelty: ClaimNovelty.CONFIRMATORY. Consensus alignment: ConsensusAlignment.NEUTRAL.

Jingyi Zhou; Yixin Liu; Yu‐Shang Yang; Pinhao Fang; Longqi Chen; Yong Yuan

doi:10.1097/js9.0000000000001696

May 29, 2024International Journal of SurgeryOpen Access

The rise of ChatGPT-4: exploring its efficacy as a decision support tool in esophageal surgery – a research letter

View Full Paper

Ask AI

Bookmark

Share

Key result

ChatGPT-4 provides commendable responses to ~97% of esophageal surgery inquiries.

n=35

Figures

Why the study?

The burden of esophageal diseases necessitates innovative solutions to enhance patient outcomes, and the efficacy of integrating Chat-GPT4 as a decision support tool in esophageal surgery required exploration.

Does Chat-GPT4 provide accurate and guideline-compliant responses to clinical decision-making questions in esophageal surgery?

Key Points

To evaluate the accuracy and clinical utility of ChatGPT-4 as a clinical decision support tool across four major esophageal disorders.
Generated responses from ChatGPT-4 (November 2023 version) for 35 clinical questions addressing esophageal cancer, Barrett's esophagus, gastroesophageal reflux disease, and achalasia of the cardia.
Three esophageal surgery specialists evaluated each response using a 0 to 3 scoring system based on clinical guidelines from the International Society for Diseases of the Esophagus (ISDE) and the National Comprehensive Cancer Network (NCCN).
Cleared model memory after each prompt to avoid data retention bias, assessing final performance via the median expert score.
Of the 35 responses evaluated, 24 (68.5%) achieved a median score of 3, 10 (28.5%) received a score of 2, and 1 (2.8%) received a score of 1, with no response scoring 0.
ChatGPT-4 achieved an acceptable or strong guidance rating (score 2 or 3) in 97.2% of inquiries, with no inter-expert score differing by more than one point for any single query.
The sole low-scoring response (score of 1) inappropriately recommended annual surveillance after five years for asymptomatic patients following esophageal cancer surgery, conflicting with updated NCCN guidelines.

Structured PICO

Does Chat-GPT4 provide accurate and guideline-compliant responses to clinical decision-making questions in esophageal surgery?

P

Population

35 clinical inquiries regarding esophageal surgery for esophageal cancer, Barrett's esophagus, gastroesophageal reflux disease, and achalasia of cardia, evaluated by three esophageal surgery specialists.

I

Intervention

Chat-GPT4 (Version: Nov 2023) used as a decision support tool, with memory cleared after each query

C

Comparator

Evaluation by three Esophageal Surgery Specialists applying guidelines from the International Society for Diseases of the Esophagus (ISDE) and the National Comprehensive Cancer Network (NCCN), alongside clinical experience

O

Outcome

Median score of Chat-GPT4 responses evaluated by three esophageal surgery specialists on a scale of 0 to 3 (where 0 signifies complete noncompliance or irrelevance, and 3 indicates nearly impeccable alignment with clinical guidelines and practices)

Chat-GPT4 demonstrated high alignment with clinical guidelines in addressing 97.2% of esophageal surgery inquiries, showing promise as a clinical decision support tool alongside necessary human oversight.

Limitations

ChatGPT does not have access to medical textbooks, journals, or real-time medical databases
May not fully understand clinical context, subtleties, or nuances involved in complex surgical procedures
Communicates exclusively through text and cannot observe surgical procedures, collaborate with a team, or physically interact with patients
Cannot substitute for human empathy, detailed communication, and understanding of nonverbal cues
Requires human oversight due to potential errors in updated guideline recommendations (e.g., postoperative surveillance frequency)
Chat-GPT does not have access to medical textbooks, journals, or real-time medical databases
Communicates exclusively through text and cannot observe surgical procedures, collaborate with a team, or physically interact with a patient
Cannot substitute for human warmth, empathy, detailed communication, and ability to respond to nonverbal cues
Potential for erroneous or outdated guideline recommendations requiring human oversight

Ask about this paper

Answers are grounded in this paper and cite their sources.

Implication

May support AI-assisted esophageal surgery queries with oversight; leaves open prospective validation before clinical adoption.

Abstract

We recently read with great interest the article titled ‘ Evaluation of Large Language Models in Breast Cancer Clinical Scenarios ’ by Deng et al . published in the International Journal of Surgery. [1] The authors provided a comprehensive analysis comparing the performance of ChatGPT-3.5, ChatGPT-4.0

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Zhou et al. (2024) conducted a letter in Esophageal disorders (esophageal cancer, Barrett's esophagus, gastroesophageal reflux disease, achalasia of cardia) (n=35). ChatGPT-4 (Version: Nov 2023) vs. ISDE and NCCN clinical guidelines evaluated by three esophageal surgery specialists was evaluated on Commendable clinical response rate (median expert score of 2 or 3 on a 0-3 scale). ChatGPT-4 provided commendable responses to 97.2% (34/35) of esophageal surgery inquiries, with 68.5% achieving the maximum median expert score of 3.

synapsesocial.com/papers/6a724270b27f15817827fc4b https://doi.org/10.1097/js9.0000000000001696

View Full Paper

Ask AI

Bookmark

Share

The rise of ChatGPT-4: exploring its efficacy as a decision support tool in esophageal surgery – a research letter

Figures

Key Points

Structured PICO

Limitations

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

The rise of ChatGPT-4: exploring its efficacy as a decision support tool in esophageal surgery – a research letter

Figures

Key Points

Structured PICO

Limitations

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading