Key result
LLaMA-RAG model outperforms XGBoost in predicting 30-day postoperative mortality with an F1-score of ~0.47.
Why the study?
Current perioperative risk prediction models overlook clinical reasoning in free-text notes, while LLM use is limited by factual accuracy concerns that retrieval-augmented generation may address.
Does integrating LLMs with RAG improve the prediction of 30-day postoperative mortality and ASA physical status classification using unstructured preoperative clinical notes compared to traditional machine learning models?
Cohort (n=24,491)
No
Does integrating LLMs with RAG improve the prediction of 30-day postoperative mortality and ASA physical status classification using unstructured preoperative clinical notes compared to traditional machine learning models?
Effect estimate: F1-score 0.4663 (95% CI 0.4654-0.4672)
Absolute Event Rate: 0.4663% vs 0.4459%
The integration of a large language model with retrieval-augmented generation using unstructured clinical notes significantly improved the prediction of postoperative mortality and ASA classification compared to traditional machine learning models.
May support LLM-RAG for note-based mortality prediction; hypothesis-generating and requires prospective validation before clinical use.
Background Accurately assessing perioperative risk is critical for informed surgical planning and patient safety. However, current prediction models often rely on structured data and overlook the nuanced clinical reasoning embedded in free-text preoperative notes. Recent advances in large language models (LLMs) have opened opportunities for harnessing unstructured clinical data, yet their application in perioperative prediction remains limited by concerns about factual accuracy. Retrieval-augmented generation (RAG) offers a promising solution—enhancing LLM performance by grounding outputs in domain-specific knowledge sources, potentially improving both predictive accuracy and clinical interpretability. Objective This study aimed to investigate whether integrating LLMs with RAG can improve the prediction of 30-day postoperative mortality and American Society of Anesthesiologists (ASA) physical status classification using unstructured preoperative clinical notes. Methods We conducted a retrospective cohort study using 24,491 medical records from a tertiary medical center, including preoperative anesthesia assessments, discharge summaries, and surgical information. To extract clinical insights from free-text data, we used the LLaMA 3.1-8B language model with RAG, using MedEmbed for text embedding and Miller’s Anesthesia as the primary retrieval source. We evaluated model performance under various configurations, including embedding models, chunk sizes, and few-shot prompting. Machine learning (ML) models, including random forest, support vector machines (SVM), Extreme Gradient Boosting (XGBoost), and logistic regression, were trained on structured features as baselines. Results A total of 520 (2.1%) patients experienced in-hospital 30-day postoperative mortality. The ASA physical status distribution was as follows: class I: 535 (2.2%); class II: 15,272 (62.4%); class III: 8024 (32.8%); class IV: 606 (2.5%); and class V: 54 (0.22%). For 30-day postoperative mortality prediction, the LLaMA‑RAG model achieved an F1-score of 0.4663 (95% CI 0.4654-0.4672), versus 0.2369 (95% CI 0.2341-0.2397) without few‑shot prompting, 0.0879 (95% CI 0.0717-0.1041) without RAG, and 0.0436 (95% CI 0.0292-0.0580) without either few‑shot prompting or RAG. Among ML models, XGBoost scored 0.4459 (95% CI 0.4176-0.4742); random forest, 0.3953 (95% CI 0.3791-0.4115); logistic regression, 0.2720 (95% CI 0.2647-0.2793); and SVM, 0.2474 (95% CI 0.2275-0.2673). For ASA classification, LLaMA‑RAG achieved a micro F1-score of 0.8409 (95% CI 0.8238-0.8551) versus 0.6546 (95% CI 0.6430-0.6796) without few-shot prompting, 0.6340 (95% CI 0.6157-0.6535) without RAG, and 0.4238 (95% CI 0.3952-0.4490) without either few‑shot prompting or RAG. In comparison, XGBoost achieved 0.8273 (95% CI 0.8209-0.8498); logistic regression, 0.7940 (95% CI 0.7671-0.7950); random forest, 0.7847 (95% CI 0.7637-0.7868); and SVM, 0.7697 (95% CI 0.7637-0.7697). Notably, the model demonstrated exceptional sensitivity in identifying rare but high-risk cases, such as ASA Class 5 patients and postoperative deaths. Conclusions The LLaMA-RAG model significantly improved the prediction of postoperative mortality and ASA classification, especially for rare high-risk cases. By grounding outputs in domain knowledge, retrieval-augmented generation enhanced both accuracy and prompt‑driven interpretability over ML and ablation models—highlighting its promise for real-world clinical decision support.
No takes yet. Share an insight, caveat, or question.
Chen et al. (2025) conducted a cohort in Patients undergoing surgical procedures under general or neuraxial anesthesia (n=24,491). LLaMA 3.1-8B language model with retrieval-augmented generation (LLaMA-RAG) vs. Machine learning models (XGBoost, random forest, logistic regression, SVM) and ablation models was evaluated on 30-day postoperative mortality prediction (F1-score) (F1-score 0.4663, 95% CI 0.4654-0.4672). The LLaMA-RAG model achieved an F1-score of 0.4663 for predicting 30-day postoperative mortality, outperforming traditional machine learning models like XGBoost.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: