Answers are grounded in this paper and cite their sources.
Background: Large language models (LLMs) are increasingly used to access organisational documentation, including standard operating procedures (SOPs), HR policies and institutional guidelines. However, retrieval-augmented generation (RAG) approaches that rely on free-form rewriting can introduce hallucinations and unstable trade-offs between completeness and conciseness, particularly in safety-and compliance-critical settings. Objectives: To evaluate extraction as a hallucination-resistant alternative to rewriting-based RAG, and to compare strategies that balance precision, recall and safety across document types and model scales. Methods: We compare multiple prompting strategies, including line-number-based source selection, extraction of relevant guideline sentences with explicit annotation of safety-related content, and a multi-stage pipeline that refines draft answers using supporting evidence from the source guideline. Constrained extraction strategies enable more reliable selection of clinically relevant source content, while direct verbatim copying provides a strong precision-oriented baseline. Experiments are conducted on documents of varying length (local NHS guidelines in acute care and oncology, as well as UK-wide NICE guidelines) and structures, using both frontier-scale and smaller locally deployable models. Performance is assessed using automatic metrics and via human expert evaluation of relevance and completeness. Results: Line-number selection provides the most consistent recall improvements for shorter, protocol-like guidelines across both large and small models, achieving medical-term recall of up to 95%. For substantially longer guidelines, draft-first extraction achieves the highest recall with selected frontier models, reaching up to 97%. Safety-oriented approaches improve precision but introduce systematic omissions, and multi-stage filtering further amplifies this trade-off. Conclusions: Line-number selection provides a robust alternative to rewriting-based RAG for accessing organisational documentation. Drafting-based approaches may offer additional benefits in very long-document settings. These findings highlight the importance of adapting retrieval strategies to document structure, model scale and the risks of omission.
No takes yet. Share an insight, caveat, or question.
Ive et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: