RAG in clinical practice: a cautionary tale of AI ‘Truthfulness’

Retrieval-augmented generation (RAG) aims to curb large language models (LLMs) hallucinations, yet its conversational reliability is uncertain. We tested a clinical RAG by executing the same query 100 times at varying dialogue lengths. The hallucination rate surged from 5%(no history) to 40% with just 10 prior exchanges, revealing a critical failure mode. Rigorous conversational testing is essential for patient safety before clinical deployment of RAG systems.