Key result
In ACLS simulations, ChatGPT achieved a median accuracy of 69% for cardiac arrest and 42% for bradycardia, demonstrating variable outputs and erroneous medication information.
Why the study?
ChatGPT's accuracy had not been evaluated in complex clinical scenarios, such as following American Heart Association ACLS guidelines for bradycardia and cardiac arrest.
Does ChatGPT accurately follow ACLS guidelines for bradycardia and cardiac arrest?
Does ChatGPT accurately follow ACLS guidelines for bradycardia and cardiac arrest?
ChatGPT showed suboptimal and variable accuracy in following ACLS algorithms for cardiac arrest and bradycardia, indicating it is not yet reliable for clinical decision support in these scenarios.
Caution against ChatGPT for ACLS decisions; leaves open its utility after targeted refinement and validation.
BACKGROUND ChatGPT is the most advanced large language model to date, with prior iterations having passed medical licensing examinations, providing clinical decision support, and improved diagnostics. Although limited, past studies of ChatGPT’s performance found that artificial intelligence could pass the American Heart Association’s advanced cardiovascular life support (ACLS) examinations with modifications. ChatGPT’s accuracy has not been studied in more complex clinical scenarios. As heart disease and cardiac arrest remain leading causes of morbidity and mortality in the United States, finding technologies that help increase adherence to ACLS algorithms, which improves survival outcomes, is critical. OBJECTIVE This study aims to examine the accuracy of ChatGPT in following ACLS guidelines for bradycardia and cardiac arrest. METHODS We evaluated the accuracy of ChatGPT’s responses to 2 simulations based on the 2020 American Heart Association ACLS guidelines with 3 primary outcomes of interest: the mean individual step accuracy, the accuracy score per simulation attempt, and the accuracy score for each algorithm. For each simulation step, ChatGPT was scored for correctness (1 point) or incorrectness (0 points). Each simulation was conducted 20 times. RESULTS ChatGPT’s median accuracy for each step was 85% (IQR 40%-100%) for cardiac arrest and 30% (IQR 13%-81%) for bradycardia. ChatGPT’s median accuracy over 20 simulation attempts for cardiac arrest was 69% (IQR 67%-74%) and for bradycardia was 42% (IQR 33%-50%). We found that ChatGPT’s outputs varied despite consistent input, the same actions were persistently missed, repetitive overemphasis hindered guidance, and erroneous medication information was presented. CONCLUSIONS This study highlights the need for consistent and reliable guidance to prevent potential medical errors and optimize the application of ChatGPT to enhance its reliability and effectiveness in clinical practice. CLINICALTRIAL
No takes yet. Share an insight, caveat, or question.
Pham et al. (2023) studied Cardiac arrest and bradycardia. ChatGPT was evaluated on Mean individual step accuracy, accuracy score per simulation attempt, and accuracy score for each algorithm. In ACLS simulations, ChatGPT achieved a median accuracy of 69% for cardiac arrest and 42% for bradycardia, demonstrating variable outputs and erroneous medication information.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: