GPT-4 provided comprehensive responses to 83.2% of heart failure-related questions compared to 78.5% for GPT-3.5, with no incorrect responses and 100% reproducibility.
Does ChatGPT provide accurate and reproducible answers to heart failure-related questions?
ChatGPT models (GPT-3.5 and GPT-4) provide accurate and highly reproducible answers to common heart failure questions, indicating potential as a supplemental patient education tool.
Absolute Event Rate: 83.2% vs 78.5%
Abstract Introduction Heart failure requires complex management with increased patient knowledge shown to improve outcomes. This study assessed the knowledge of Chat Generative Pre-Trained Transformer (ChatGPT) and its appropriateness as a supplemental resource of information for patients with heart failure. Materials and Methods A total of 107 frequently asked heart failure-related questions were included. Responses were generated using GPT-3.5 and GPT-4. The accuracy and reproducibility of responses were graded by two reviewers board-certified in cardiology, with differences resolved by a third reviewer board-certified in cardiology and advanced heart failure. Accuracy was graded using a four-point scale: 1) Comprehensive 2) Correct but inadequate 3) Some correct and some incorrect 4) Completely incorrect. Results GPT-4 displayed a greater proportion of comprehensive knowledge for the categories of “basic knowledge” and “management” (89.8%, 82.9%). GPT-3.5 performed best in the “management” and “other” category (prognosis, procedures, and support) (78.1%, 94.1%). There were two total responses (1.9%) graded as “some correct and incorrect” for GPT-3.5, while no “completely incorrect” responses were produced. The models also provided highly reproducible responses, with GPT-3.5 scoring above 94% in every category and GPT-4 with 100% for all answers. Conclusions GPT-3.5 and GPT-4 answered the majority of heart failure-related questions accurately and reliably. ChatGPT may lead to better outcomes in patients with heart failure by providing accessible and easy to understand health education. However, in its current state the model necessitates further rigorous testing and validation to ensure patient safety and equity across all patient demographics.
King et al. (Sat,) conducted a other in Heart failure (n=107). GPT-4 vs. GPT-3.5 was evaluated on Comprehensive accuracy of responses. GPT-4 provided comprehensive responses to 83.2% of heart failure-related questions compared to 78.5% for GPT-3.5, with no incorrect responses and 100% reproducibility.