Retrieval-augmented Large Language Models for automated discharge summary generation in post-surgical ICU cancer patients

Introduction Discharge summary generation is an essential yet time-consuming component of clinical documentation, particularly for post-surgical cancer patients admitted to the Intensive Care Unit (ICU), where clinically relevant information is distributed across structured electronic health records (EHRs) and unstructured clinical notes. Although Large Language Models (LLMs) have shown promise for automating this task, they remain susceptible to hallucinations and often struggle to incorporate previously unseen patient information. Retrieval-Augmented Generation (RAG) has been proposed to address these limitations by grounding text generation in retrieved clinical evidence. Methods This study separates the contribution of retrieval from that of fine-tuning using a paired two-by-two design: three open-source LLMs: Qwen2.5-7B-Instruct, StableLM-Zephyr-3B, and DeepSeek-R1-Distill-Qwen-7B, were each evaluated as pretrained and parameter-efficient fine-tuned models, with and without a hybrid RAG pipeline combining sparse (BM25) and dense (FAISS) retrieval, giving 12 configurations. Using structured data from the MIMIC-IV dataset and two disclosed note-derived fields, all configurations generated multi-section discharge summaries for the same fixed evaluation set of 50 patients and were assessed with ROUGE, BLEU, and BERTScore, with bootstrap confidence intervals and paired per-patient comparisons. Results Fine-tuning improved every model on every metric, with confidence intervals excluding zero, and fine-tuned summaries matched the length of physician-authored references; the smallest model (StableLM-Zephyr-3B) achieved the highest ROUGE-1 (0.278) after fine-tuning. Retrieval augmentation, implemented as style-reference retrieval from training cases, produced no measurable change in this setting for either pretrained or fine-tuned models. Discussion These findings indicate that small, parameter-efficient fine-tuned open-source models offer a practical option under constrained compute for automated ICU discharge summary generation, and that the value of retrieval augmentation should be demonstrated against fine-tuned baselines rather than assumed.