GPT-4 Turbo achieved F1 scores of 0.86 (95% CI, 0.83-0.89) for hospital mortality and 0.81 (95% CI, 0.78-0.83) for ICU admission, but performed poorly on numerical duration prediction tasks.
Observational
Yes
Can the large language model GPT-4 Turbo accurately predict perioperative risk and postoperative outcomes using electronic health record notes?
General-domain large language models like GPT-4 Turbo show promise for perioperative risk stratification on classification tasks such as mortality and ICU admission, but are currently inadequate for predicting numerical duration outcomes.
Importance: General-domain large language models may be able to perform risk stratification and predict postoperative outcome measures using a description of the procedure and a patient's electronic health record notes. Objective: To examine predictive performance on 8 different tasks: prediction of American Society of Anesthesiologists Physical Status (ASA-PS), hospital admission, intensive care unit (ICU) admission, unplanned admission, hospital mortality, postanesthesia care unit (PACU) phase 1 duration, hospital duration, and ICU duration. Design, Setting, and Participants: This prognostic study included task-specific datasets constructed from 2 years of retrospective electronic health records data collected during routine clinical care. Case and note data were formatted into prompts and given to the large language model GPT-4 Turbo (OpenAI) to generate a prediction and explanation. The setting included a quaternary care center comprising 3 academic hospitals and affiliated clinics in a single metropolitan area. Patients who had a surgery or procedure with anesthesia and at least 1 clinician-written note filed in the electronic health record before surgery were included in the study. Data were analyzed from November to December 2023. Exposures: Compared original notes, note summaries, few-shot prompting, and chain-of-thought prompting strategies. Main Outcomes and Measures: F1 score for binary and categorical outcomes. Mean absolute error for numerical duration outcomes. Results: Study results were measured on task-specific datasets, each with 1000 cases with the exception of unplanned admission, which had 949 cases, and hospital mortality, which had 576 cases. The best results for each task included an F1 score of 0.50 (95% CI, 0.47-0.53) for ASA-PS, 0.64 (95% CI, 0.61-0.67) for hospital admission, 0.81 (95% CI, 0.78-0.83) for ICU admission, 0.61 (95% CI, 0.58-0.64) for unplanned admission, and 0.86 (95% CI, 0.83-0.89) for hospital mortality prediction. Performance on duration prediction tasks was universally poor across all prompt strategies for which the large language model achieved a mean absolute error of 49 minutes (95% CI, 46-51 minutes) for PACU phase 1 duration, 4.5 days (95% CI, 4.2-5.0 days) for hospital duration, and 1.1 days (95% CI, 0.9-1.3 days) for ICU duration prediction. Conclusions and Relevance: Current general-domain large language models may assist clinicians in perioperative risk stratification on classification tasks but are inadequate for numerical duration predictions. Their ability to produce high-quality natural language explanations for the predictions may make them useful tools in clinical workflows and may be complementary to traditional risk prediction models.
Chung et al. (Wed,) conducted a observational in Patients undergoing surgery or procedure with anesthesia. GPT-4 Turbo large language model vs. Various prompting strategies (original notes, note summaries, few-shot prompting, chain-of-thought) was evaluated on F1 score for binary and categorical outcomes; Mean absolute error for numerical duration outcomes. GPT-4 Turbo achieved F1 scores of 0.86 (95% CI, 0.83-0.89) for hospital mortality and 0.81 (95% CI, 0.78-0.83) for ICU admission, but performed poorly on numerical duration prediction tasks.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: