HoneyMark: Protecting Data Integrity in RAG via Synthetic Trap Documents and Post-hoc Watermarking

Data owners lack effective methods to detect unauthorized incorporation of their content into Retrieval-Augmented Generation (RAG) systems. Existing approaches likeWARD (Watermarkbased RAG Dataset inference) compromise data integrity by modifying original documents with watermarks prone to signal loss. We propose HoneyMark, a trap document-based approach that achieves dataset membership inference without altering original data. HoneyMark generates synthetic documents with POSTMARK post-hoc watermarks and inserts them as traps into datasets. When queried about synthetic content, RAG systems containing these documents reveal watermark vocabulary in responses, providing statistical evidence of unauthorized usage. Our fourdimensional generation framework creates trap documents semantically consistent with original data while maintaining content uniqueness. Experimental results demonstrate perfect discrimination (AUC-ROC = 1.0) with only 16 queries and 100% TPR at 1% FPR with 20 queries, significantly outperforming existing methods while preserving data integrity.