The BERT-DNN model incorporating unstructured clinical text achieved an AUROC of 0.964 for predicting 30-day postoperative mortality, significantly outperforming logistic regression and ASAPS.
Cohort (n=121,313)
No
Does a BERT-DNN model incorporating unstructured clinical text improve the prediction of in-hospital 30-day postoperative mortality compared to models using only structured data in patients undergoing surgery?
Incorporating unstructured clinical text via a BERT-DNN model significantly improves the precision of predicting 30-day postoperative mortality compared to models using only structured data.
Effect estimate: AUROC 0.964 (95% CI 0.961-0.967)
Absolute Event Rate: 0.964% vs 0.952%
p-value: p=0.0005
BACKGROUND: Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text. OBJECTIVE: This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed. METHODS: We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05. RESULTS: The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203). CONCLUSIONS: Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.
Chen et al. (Tue,) conducted a cohort in Postoperative mortality (n=121,313). BERT-DNN model (incorporating unstructured clinical text) vs. Logistic regression, ASAPS, DNN, Random forest, and XGBoost models was evaluated on In-hospital 30-day postoperative mortality prediction (AUROC) (AUROC 0.964, 95% CI 0.961-0.967, p=0.0005). The BERT-DNN model incorporating unstructured clinical text achieved an AUROC of 0.964 for predicting 30-day postoperative mortality, significantly outperforming logistic regression and ASAPS.